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PREFACE 



Rand is conducting, under the sponsorship of the U.S. Office of Education, a 
several-year study of federally funded programs designed to iniroduce and spread 
innovative practices in public schools. These change agent programs normally offer 
temporary federal funding to school districts as "seed money." If an innovation is 
successful, it is assumed that the district will continue and disseminate part or all 
of the project using other sources of funds. The Rand study examines four such 
federal change agent programs— Elementary and Secondary Education Act Title III, 
Innovative Projects; Elementary and Secondary Education Act Title VII, Bilingual 
Projects; Vocational Education Act, 1968 Amendments, Part D, Exemplary Pro- 
grams; and the Right-To-Read Program. The study identifies what tends to promote 
various kinds of changes in the schools and what doesn't; in particular, the Rand 
study will identify for federal, state, and local policymakers the nature, permanence, 
and extent of dissemination of innovations that are associated with the various 
federal programs and with various federal, state, and local practices. 

A series of five reports describes the results of the first year of the Rand study 
(July 1973 to July 1974). Volume I (R-1589/1-HEW, A Model of Educatioruil Change) 
provides a theoretical perspective for the Rand study by analyzing the current state 
of knowledge of planned change in education and by proposing a conceptual model 
of factors affecting change processes within school districts.^ 

This report. Vol. II of the series, contains the analysis of survey data collected 
from a national sample of 293 projects in 18 states during November and December 
1973. 

Volume III (R-1589/3-HEW, The Process of Change) summarizes the results of 
29 case studies of change agent projects conducted by Rand staff members and 
consultants in 25 school districts during April and May 1974. These case studies 
were chosen from the original sample of 293 projects initially surveyed. Volume III 
also describes the role of state education agencies in choosing and disseminating the 
change agent projects. 

Four technical appendixes to Vol. Ill describe in detail the federal program 
management approach, state education agency participation, and case studies for 
each of the programs in the study: Title III, App. A; Reading, App. B; Bilingual 
Education, App. C; and Career Education, App. D. Appendix A should be of particu- 
lar interest to researchers or practitioners concerned with the inttoduction of new 
approaches to classroom instruction. 

Volume IV (R-1589/4-HEW, The Findings in fieuicu;J summarizes the findings 
of Vols. I, II, and III, and also synthesizes extensive data collected by Rand on 
federal-level program strategy and management for each of the change agent pro- 
grams. Volume IV also includes a discussion of alternative federal strategies for 
promoting innovation. 

' Because of Rand's interest in advancing knowledge of organizational behavior in educational insti- 
tutions, the research underlying this report was supported in part by an allocation of Rand corporate 
research funds. 
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Volume V (R-1589/5-HEW,, Executive Summary) presents a distillation of the 
study's methods and results for a general audience. 

Subsequent research will collect additional data on Titles III and VII of ESEA, 
with particular focus on projects whose federal funding has expired. 



SUMMARY 



Xlhis second volume in the change agent series reports the interim results of an 
e;cplW;atory statistical analysis of a survey of a nationv^^ide sample of 293 change 
agent^ projects funded by the following federal demonstration programs: 

Title III, ESEA (State and Federal) 

Vocational Education, 1968 Amendments, Part D, Exemplary Programs 

(Federal and State) 
Title VII, ESEA, Bilingual Education 
Right-To'Read 

The study pddresses three research questions: 

• To what extent did differences among the federal change agent programs 
explain variations in the implementation and continuation of innovative 
projects? 

• Which characteristics of innovative projects significantly affected their 
implementation and continuation? 

• Were differences in institutional settings related to variations in project 
implementation and continuation, controlling for other factors, and, if so, 
which institutional aspects had significant and important effects? 



DATA ANALYSIS PLAN 

■*> 

An explanatory model or series of assumptions and hypotheses about school 
district decisionmaking forms the foundation for the data analysis. The model im- 
plies that two levels of analysis should be pursued— the school /classroom level and 
the school district level. At the classroom level, implementation is the primary 
concern; at the school district, expected continuation is appropriate for analysis. 

For each of these levels, the data analysis involved two steps. First, we selected 
and measured the most important dependent and independent variables implied by 
. the central research questions. In particular, the independent variables measure 
important project characteristics, aspects of the institutional setting, and federal 
program inputs; the dependent variables measure implementation outcomes and 
expected continuation. 

Second, the effects of an independent variable on the outcome measures were 
estimated using multiple regression techniques. This statistical approach allowed us 
to test the significance of the variables as well as suggesting the relative importance 
of the independent variables in explaining variations in project outcomes. 

Dependent Variables 

We measured, using teacher responses, four types of implementation outcomes 
at the classroom level— the perceived success of the project (percentage of goals 
achieved), the fidelity of implementation jthe extent to which the project was imple- 
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mented as laid out), the extent of teacher change, and the didiculty of implementa- 
tion. At the district level, superintendents* answers were used to measure expected 
continuation of the project, in whole or in part, after the end of federal funding. 

Independent Variables 

At the classroom level, we considered implementation outcomes to depend on 
project characteristics, aspects of the institutional setting, and the federal change 
agent program sponsoring the project. Project characteristics were divided into 
three components — project resources, educational method or treatment, and im- 
plementation strategies — and for each component operational variables were meas- 
ured. Similarly, the institutional setting was categorized into organizational cli- 
mate, school /classroom characteristics, and attributes of the principal, actors. 

At the district level, we conceived of the decision to continue an innovation to 
be a function of both the input variables — the educational treatment or methods and 
the different federal change agent management strategies— and factors reflecting 
political-economic and organizational concerns— the project*s past success, the cen- - 
trality or importance of the educational needs served by the project, the resources 
required by the project, and the organizational-political forces inhibiting or promot- 
ing* the innovation. 

FEDERAL PROGRAM COMPARISONS 

The program guidelines and management strategies of the various federal 
change agent programs were briefly reviewed, and their differences from each other, 
which are marked, were noted. We examined the extent to which the federal pro- 
grams had affected project design choices during the initiation phase of an innova- 
tion, difficulties arising during implementation, implementation outcomes at the 
classroom level, and continuation. 

Program Effects on Project Design 

We found that the guidelines, program characteristics, funding, and priorities 
of federal programs did affect project design, but only in a partial way. Whether local 
districts were seeking opportunities for federal money or were attempting to solve 
their problems by initiating innovative projects, some educational methods were 
more likely to be employed than others because some federal programs— either in 
their administrative guidelines or in their focus—- fostered these methods. Thus, in 
the absence of the Right-To-Read program, locally initiated reading projects might 
be less likely to adopt behavioral modification techniques and concentrated tradi- 
tional staffing. For Vocational Education, the existence of a federally funded pro- 
gram—and the funding opportunities it afforded— induced substantial enrichment 
activities that may by pursued only as long as the federal money exists. On the other 
hand, federal funding opportunities appeared to have less eflect on such locally 
conceived projects as innovations in classroom organization. 

Thus, federal policy, using as its instruments guidelines and categorical priori- 
ties tied to different types of funding, may have limited leverage on the design 
choices made by local school districts. In particular, the federal programs can influ- 



Vll 



ence how intensively some educational treatments are tried. However, federal policy 
instruments of the type employed by the change agent programs may have little 
influence on how frequently local districts initiate some types of educational meth- 
ods, particularly classroom organization change, in order to meet their own needs 
and priorities. 

Program Effects on Difficulty of Implementation 

The problems that most seriously affected implementation stemmed from two 
sources. First, innovation is intrinsically a disruptive process. Problems attributable 
to inexperience of project participants in planning for change and adapting to its 
demands were bound to arise. Second, projects encountered difficulties that reflected 
the selection mechanisms, administrative guidelines, and substantive priorities of 
the federal programs. In particular: 

1. State Title III projects experienced difliculties because of insufficiently 
defined goals and inadequate prior planning. 

2. Federal Title HI projects had "top-down^' problems of complicated or un- 
clear techniques and required more additional work than teachers could 
handle. 

3. Title VII Bilingual projects lacked bilingual materials and staff. 

4. Vocational Education projects experienced difficulties gaining support 
within the district and within the community. 

5. Right-To-Read projects had difficulties of implementation that arose from 
the teachers' unfamiliarity with prescribed materials and methods, and 
from adaptation requirements that were not anticipated by the program 
management strategy. When parental opposition existed, Right-To-Read 
projects experienced considerable difficulty. 

Thus, the patterns of difficulty associated with the implementation of different 
federal programs demonstrate that the management strategies have subtle influ- 
ences beyond the initiation stage. Insofar as federal programs affect initial project 
design choices, they also may create specific barriers that innovations must over- 
come. 

Program Effects on Implementation Outcomes 

The federal change agent programs did noi differ significantly from each other 
on the average implementation outcomes of projects that they fund. That is, the 
differences between programs explained little of the variation in project outcomes. 
Nor did variation in educational treatment for projects within the federal programs 
account for much more of the oiftcome variability. When other factors involving the 
institutional setting, implementation strategies, and project resources were statis- 
tically held constant, several marginally significant effects of program differences 
emerged: 

1. Vocational Education projects had the highest average reported goals 
achieved but were the least likely to induce teacher change and were the 
easiest to implement. 

2. Right-To-Read projects and Bilingual, Title VII projects were less likely to 
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produce teacher change than Title III projects. Bilingual projects were 
relatively difficult to implement. 

Program Effects on Continuation 

The findingo for expected continuation parallel those for implementation out- 
comes. The differences between federal change agent programs explained little of 
the variation in the planned continuation of projects. This general conclusion held 
true after the introduction of variations in educational treatments within federal 
programs as well as afler controls for other independent variables. 

However, some federal program differences were marginally significant: 

1. Vocational Education projects were the least likely to be continued even 
though superintendents viewed them as accomplishing a high percentage 
of their goals. Vocational Education projects seemed to achieve a high 
percentage of their goals because they were not attempting major innova- 
tions but simply added career enrichment materials to the curriculum. 
Data from the superintendents suggest that these add-on projects were the 
least likely to be continued by the district when federal funds terminated. 

2. Title VII, Bilingual projects were viewed as the most difficult to implement 
and as having achieved the lowest percent^ige of their goals. Nonetheless, 
their expected continuation was significantly higher than the average of 
projects funded by the other programs. 

3. Superintendents reported Right-To-Read projects as relatively the least 
difficult to implement and somewhat more likely to be continued than 
Vocational Education projects. 

4. Title III projects with high levels of classroom organization changes were 
more likely to be continued than other Title III projects, even though they 
were more difficult to implement and not perceived as more successful. 

5. The expected continuation and perceived success of Bilingual projects de- 
pended on the mix of methods employed. High levels of behavioral modifi- 
cation instruments and of classroom organization changes were seen as 
difficult in this program, as relatively unsuccessful, and as less likely to be 
continued. In contrast, intensive use of traditional staffing and enrichment 
tended to contribute to perceived success and to expected continuation. 



FACTORS AFFECTING IMPLEMENTATION OUTCOMES 

The following results were obtained from a multiple regression analysis of the 
factors affecting project implementation outcomes at the classroom level: 

1. The effective implementation of innovative projects depended primarily 
on a supportive institutional setting and on an implementation strategy 
that fostered the mutual adaptation of the staff to the project's demands 
and of the project's design to the reality of its setting. 

2. Projects funded by tne same federal program showed considerable varia- 
tion in their implementation strategies and institutional settings. These 
within-program variations affected project implementation more signifi- 
cantly than did the differences between federal pro-ams. 
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3. Projects using similar educational methods or technologies varied consid- 
erably in their implementation strategies and institutional settings. These 
variations affected project implementation more significantly than did the 
differences between the ^educational methods or technologies themselves. 

4 Superintendents, who tend to be organizationally remote, provided a gen- 
eralized support that may have made schools receptive to innovations; such 
receptivity may be essential to Title III projects. Elementary school princi- 
pals appear to have been "gatekeepers" of change, either facilitating or 
inhibiting implementation. 

5. The following elements of implementation strategies promoted teacher 
change: 

a. Staff training, 

b. Frequent and regular meetings. 

c. Staff meetings held in conjunction with staff training. 

d. The quality and amount of change required by the project. 

The fbllowing elements of implementation strategies inhibited perceived 
success or teacher change: 

a. The lack of the above elements. 

b. Teachers not participating in day-to-day implementation decisions. 

c. For Title III projects, the lack of local material development. 

The following elements of implementation strategies were not significant- 
ly related to implementation outcomes: 

a. The quantity of planning. - 

b. Participants* freedom to alter the basic project design on difficult pro- 
jects. 

c. Part or full pay for training. 

6. Elementary school projects were more effectively implemented than jun- 
ior or senior high school projects and were more likely to produce teacher 
change. 

7. Within the range of variation of differential funding considered here, a 
project*s funding level did not have significant effects on teacher change or 
perceived stccess. Projects that serve most of, or the entire, student body 
of elementary schools were unlikely to produce teacher change. The more 
concentrated were the resources of Title III projects, the more likely was 
teacher change to occur. 



FACTORS AFFECTING EXPECTED CONTINUATION 

The results of a multiple regression analysis of the factors affecting continua- 
tion indicated that the district decision to continue a project was based on the 
superintendents weighing of four general concerns: the project s past success, the 
centrality or importance of the educational need served by the project, the resources 
required by the project, and the organizational-political forces inhibiting or promot- 
ing the innovation. ^ r) 
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In particular, the following detailed hypotheses proved statistically significant: 

1. The higher the perceived success, the more likely the project was to be 
continued. 

2. Projects that were central to the educational goals of the district were 
more likely to be continued. 

3. The more expensive the project, che less likely was its continuation. 

4. The larger the district, the more likely was project continuation. 

5. The higher the percentage of blacks in the district population, the less 
likely was continuation. (This finding was not significant for our sample of 
Title III projects.) 

6 The higher the percentage of Spanish-speakiiig population in the district, 
the more likely was continuation. This positive effect of percentage of 
Spanish on continuation reflected the likely continuation of Bilingual pro- 
jects whether funded by Title VII or Title III. 

7. General political-social difficulties failed to be significant (perhaps because 
of poor measurement) but difficulty from student test scores— a sensitive 
indicator of community pressure — was positively related to continuation. 

8. Districts that initiated projects with the prior intention of continuing were 
likely to continue their projects, all other things being equal (e.g., resources 
and success). Suci. prior intention may be indicative of a problem-solving 
approach on the pa.* jf the district. 

9. Projects supported by principals were likely to be continued. 

10. The longer the^uperintendent*s tenure in the district, the more likely 
projects were to be continued. 
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L INTRODUCTION 



This volume describes and identifies major factors affecting the outcomes of 
innovations attempted by school districts using federal funds. The analysis,. which 
quantitatively examines data collected by a nationwide survey of 293 change agent 
projects, is one component of a Rand study sponsored by the U.S. Office of Education. 



BACKGROUND OF THE CHANGE AGENT STUDY 

During the 1950s and 1960s, two important initiatives, the National Defense 
Education Act of 1958 and the Elementary and Secondary Education Act of 1965, 
defined a new federal role in local education— large-scale support of federally man- 
dated programs aimed at specified goals: subsidizing special curriculum develop- 
ments, educating the disadvantaged, training young people for careers, broadening 
access to higher education, and encouraging innovation in the public schools, which 
is the subject of this study. Approximately 10 percent of the federal aid to public 
sr^ools, currently exceeding $3.5 billion annually, is aimed at promoting education- 
al innovations. These funds are spent, primarily by the U.S. Office of Education, 
through a number of avenues and ^ith diverse strategies, including the so-called 
change agent programs analyzed in this study:* Elementary and Secondary Educa- 
tion Act Title III, Innovative Projects ($150 million annually); Elementary and 
Secondary Education Act Title VII, Bilingual Projects ($45 million); Vocational^ 
Education Act. 1968 Amendments, Part D, Exemplary Programs ($16 million); and 
Right-To-Read ($12 million). 

Each change agent program has a distinct focus and management strategy. The 
largest of the programs considered here. Title III, is designed to improve the quality 
of public education both by introducing model practices that are new to American 
education and by spreading existing successful'practices to schools that are not 
aware of them. The competition for Title III grants of three-year duration is open 
to almost any kind of project that local schools wish to propose. In 1973-1974, the 
first year of our study, 15 percent of Title III money was granted directly to local 
education agencies (LEAs) by the Office of Education, and the remaining was allocat- 
ed to state education agencies (SEAs), who in turn made grants to LEAs. Other 
federal change agent programs are more narrowly targeted and have more specific 
/funding criteria. Right-To-Read represents aiHrtfempt by the,Office of Education^to 

/ create a national educational priority for reading, particularly for disadvantaged 
students. The Right-Tg-Read demonstration projects, the program component ad- 
dressed in this study, included a prescribed pljanning and management strategy in 
an eflbrt to facilitate effective implementation. Vocational Education, Part D, was 

/ designed to create exemplary programs to enhance career awareness and readiness. 
Congress, believing that many SEAs were not able to promote significant innova- 

» Other federal programs, not studied here, also aim at encouraging' innovations, for example, cert^iin 
programs for handicapped student^j, experimental schools, educational voucher demonstrations. Follow 
Through, elements; of the Emergency School Assistance Act, 
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tions, gave USOE the authority to fund local projects Hirectlj through the Part D 
program. Half of the Part D appropriations are allotted to the SEAs, the other half 
to USOE. Title VII (Bilingual Education) originMy sought to provide model projects 
for the special needs of children whose English-speaking ability was limited. The 
program has subsequently also developed into an effort to maintain and encourage 
"cultural pluralism'' in American public education, with strong political support 
from many people of Spanish-language origin. 

, Despite these differences in focus and management strategy, the change agent 
programs have a common purpose, the stimulation and spread of educational inno- 
vations. They also have a common policy instrument, the provision of temporary 
funds (3 to 5 years), which, although small relative to the budget of a school district 
(ranging from grants of $10,000 or less to several hundred thousand dollars per 
year), are intended to fund new educational services, not to support existing prac- 
tice. These programs also rest on common assumptions. They all assume more or less 
explicitly that American education should be doing better in respect to a variety of 
goals ranging from specific objectives, such as student reading achievement, to the 
broad concerns of student personality and social development. Moreover, the change 
agent programs assume that educational practices, procedures, and methods can be 
improved witlyfi the existing educational structure. Federal policy presumes that 
providing funds to a relatively small number of districts to try innovations will 
demonstrate the value of some of these innovations, which will then be adopted 
selectively by other districts. 

Numerous studies have evaluated, described, or analyzed innovative projects 
and federal programs designed to stimulate educational change.^ These studies are 
mixed both in their research quality and in their findings. Anecdotal J^nd single-case 
study evaluations usually claim considerable ''success'* for projects supported by 
federal funds. Yet quantitative evidence and careful fieldwork suggest something 
quite different: 

• Variations in student outcomes have not been consistently related to 
variations in treatments, once nonschool factors are held constant. 

• "Successful" projects have lacked stability and have not been easy to "ex- 
port" from school to school or district to district. 

The alleged ineffectiveness and instability of innovative projects might be dismissed 
either as premature — it may indeed be too soon to judge innovations that take many 
years to develop~--or as subject to measuring the wrong student outcomes in the 
wrong ways—profound measurement problems surely plague assessments in this 
field. However, it is more prudent to assume that present policies and the assump- 
tions underlying them need to be reexamined and, where appropriate, revised and 
redirected. With these objectives in mind, USOE asked Rand to undertake a several- 
year study to examine innovative projects funded by the various change agent 
programs,-' and, in particular, to treat the following questions: 

1. How should the degree, quality, and extent of innovation and dissemina- 
tion of change in the public schools be assessed? 

* Volume I of this study reviews the literature. 

^ In this report, we refer to pro^mtm when describing the federal change agent initiatives, for 
example. Right-To-Read. We refer to projc(^ts when dcbcribing the particular innovation belected by a 
school district. 
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2. How do school districts select, introduce, implement, incorporate, and 
spread different kinds of innovations? 

3. How do differences in target groups, resource use, educational treatment, 
project strategies, and other characteristics affect the initiation, implemen- 
tation, persistence, and dissemination of innovations? 

4. How do differences in institutional and political contents affect the initia- 
tion, implementation, persistence, and dissemination of innovations? 

5. How, if at all, do differences in the relations among the characteristics of 
innovations and ihe institutional-political setting affect the initiation, im- 
plementation, persistence, and dissemination of innovations? 

6. Do the different federal change agent program strategies have different 
effects on the initiation, implementation, and outcomes of local projects? 

7. What should federal policies be toward educational innovation in light of 
the political, financial, and organizational constraints that the federal gov- 
ernment faces in its dealings with the public schools? 



CENTRAL PREMISES AND THEORETICAL FRAMEWORK 

To study the questions listed above, we developed a theoretical framework— a 
set of hypotheses and assumptions— that has guided the research. The framework 
rests on two assumptions: first, that there are distinctively different stages of the 
innovative process and, second, that the institutional setting profoundly influences 
the nature and impacts of an educational innovation, as well as its likely perma- 
nence and dissemination."* 

Stages of Innovation 

We view the change or innovative process as consisting of three stages: initia- 
tion, implementation, and incorporation. The initiation stage in the life of an in- 
novative project occurs when local school officials conceive and formulate plans, seek 
resources, and make decisions about which projects they should select and support. 
We hypothesize that the support and commitments made in the initiation period 
affect what happens when project implementation begins. 

In the implementation stage, the project confronts the reality of its institutional 
setting, and project plans must be translated into practice. Many innovative projects 
fail or are disappointing because they are not implemented according to plan. But 
the issue of implementation is often more subtle and complicated than mere fidelity 
to predesigned means for attaining specified educational goals. We hypothesize that 
as a consequence of the institutional characteristics of local school systems the 
implementation of those educational innovations that result in significant change 
intrinsically involves a process of mutual adaptation.^ Thus, the initial design of an 

* The theoretical framework is justified and elaborated in Vol. I. 

^ Volume I defines an innovation as a plan with a statement of goals and means designed to change 
sUindard behavior, practices, or procedures. Many educationaj mnovations tend to have abstract goals, 
to lack specific, and clarity of means, and to have considerable uncert^iinty as to the relationship 
between means and ends. Such uncertainty makes it desirable for the innovation to become developed, 
revised, or, in short, adapted to the realities of its mstitutional setting Accordingly, we define implemen- 
tation as the change process that occurs when an innovative project impinges a{>on an organization 
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in! ivative project becomes adapted to the particular organizational setting of the 
school, classroom, or other institutional hosts; at the same time, the organization 
and its members adapt to the demands of the project. Therefore, many educational 
innovations may fail to have desirable effects because the project is not adapted to 
the institutional setting or vice versa. 

The term incorporation is used to denote the final stage in innovation — when 
an innovative practice may lose its ''special project" status and become part of the 
routinized behavior of the LEA. Incorporation represents the most serious commit- 
ment on the part of the district, as federal ''seed money" is withdrawn and decisions 
must be made about not only tr/ie(/ier but also what components of^ and on what scale 
a project should be continued within the district. We believe this decision may 
involve more than the success or failure of the project during its trial period. Eco- 
nomic, political, and organizational pressures and constraints may play major roles 
in determining the innovation's future. 

Not only do these three stages involve somewhat different activities and deci- 
sions, but the significance of actors and issues also changes from one stage to anoth- 
er. Thus we believe that a key to designing and assessing federal, state, and local 
policy lies in understanding how the stages of innovation work in different locations, 
for different innovations, and for the various change agent programs. 



Factors Affecting "Outcomes" ^^|^ 

The change process itself i^^^^K^nt, but systematic analysis also should 
identify major factors affecting tU B icomes" or effects of innovations. The first 
research problem is to select or devise^itable measures of outcomes. Since federal 
programs and local and state policies aim at improving the education of children, 
the most direct and natural measure of the effectiveness of an innovative project 
would be changes in behavior, attitudes, or test scores of students. However, for the 
purposes of the study, it was both premature and inappropriate to measure student 
outcomes. The innovations were generally new to the district and consequently 
require an extended period before their true effects can be assessed. Moreover, these 
projects differed considerably in their focus, goals, methods, and assessment proce- 
dures. Therefore, there was no practical way to compare, for example, outcomes of 
bilingual reading projects with projects dealing with remedial mathematics.® More 
important, innovations may not be implemented according to plan. The first issue 
before assessing longer range student impacts is thus to measure the effectiveness 
of the project's implementation. 

We studied three types of effectiveness measures or "outcomes": 

1. Implementation: (a) the relative extent to which project goals are 
achieved, (b) the type and extent of change in teacher behavior, and (c) the 
extent to which the project as proposed compares with the project as imple- 
mented. 

2. Continuation: the extent to which the project is continued by the LEA 
afler federal funds are withdrawn. 

3- Dissemination: the extent to which the project is diffused to other schools 
in the district or to other districts. 

* Volume I of this study reviews the literature on the measurement of student outcomes. 
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The change agent study consists of two phases. In phase one, which is covered 
in the first five volumes of this series, we deal with projects that have not yet 
completed their period of federal funding; consequently, the analyses here are limit- 
ed to implementation outcomes and expected continuation (plus some preliminary 
information about dissemination).*' 

We believe that three interrelated sets of factors affect these outcomes: 

• Characteristics of projects 

• Federal and state policies 

• Institutional settings 

Most project evaluations investigate only the relationship between outcomos and 
the educational technology or method that is characteristic of the project. Yet two 
# projects with the same educational method can be implemented in quite different 
ways, leading to different outcomes. This means that projects also can be character- 
ized by their implementation strategies, that is, the design decisions about what 
should be done to implement the project, how it should be done, and by whom. We 
hypothesize that implementation strategies will have major effects on project out- 
comes. 

Because project evaluations usually are case studies of LEA projects funded by 
the same federal change agent program, they tend to treat federal policy unsys- 
tematically. This study examines four federal programs, each with a different aim 
and different management strategies. This provides an opportunity to compare 
innovative projects funded by different programs and an opportunity to assess the 
extent to which project outcomes are explained by differences between the federal 
programs. 

The most serious omission of most evaluation literature is its failure to take into 
account the institutional setting in which innovations operate. Such major elements 
of the institutional setting as demographic, economic, and political conditions of the 
district, community influences, student characteristics, and organizational char- 
acteribtics are usually neglected. Although the analysis will examine all relevant 
institutional elements wherever possible, the research design and analysis have 
been predicated v.n our belief that organizational characteristics of LEAs have major 
impacts on innovations and that federal, state, and local policies largely ignore these 
characteristics. 



OVERALL RESEARCH PLAN 

The theoretical framework thus suggests that two areas need to be studied if the 
broad questions implied by federal change agent policy are to be examined system- 
atically: first, the process of change and, second, the factors (project characteristics, 
institutional setting, and ffed'eral and state policies) affecting the innovative project 
Accordingly, the first phase of Rand's research efforts consisted of two principal data 
collection activities (App. A to Vol. IV provides a more detailed description of the 
overall study design): 

' The Federal Vocational Education projects (9) and halfof the State Vocational Education projects 
( 14) were the only projects in the sample that had completed their federal funding before the beginning 
of our research. 
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1. A nationwide survey of 293 change agent projects in the last or next to last 
year of federal funding was administered in November and December 1973 
and January 1974 by the National Opinion Research Center of the Univer- 
sity of Chicago, under a subcontract with Rand. The survey, which included 
personal interviews with project participants at all levels of the LEA, was 
designed to elicit information about factors affecting a sample of innovative 
projects funded by the various federal change agent programs. 

2. After the survey was completed, during April and May 1974, the Rand 
staff conducted 29 field studies of projects from the survey sample. The field 
studies, consisting of observations and interviews in project schools, were 
designed to explore and compare the process of change m a number of 
institutional settings for innovative projects with different characteristics, 
funded by the various change agent programs.^ . 

Because data collection had to be completed within the 1 973-74 school year, the 
survey and field studies could not be fully integrated (in the sense that the results 
of the survey would establish hypotheses and important research avenues that the 
fieldwork would then explore in depth). However, the two efforts were designed to 
complement and reinforce each other. The survey's quantitative information about 
structural factors affecting outcomes has provided a general perspective for the 
fieldwork's qualitative, small-sample information about process; the fieldwork's in- 
depth analysis has added realistic detail that has been helpful in interpreting the 
survey data. In short, the two ef^prts tend to validate each other. 

The second phase of the change agent study, drawing on the preliminary 
findings and the data of the initial year, will focus on describing what happens when 
the federal funding for the innovative project is terminated. Accordingly, the Rand 
staff will revisit projects in .the first-year sample that were in their last year of 
federal funding. Thus, we will be able to examine hypotheses about continuation and 
dissemination that are raised by the first year's findings. 



SURVEY AND DATA ANALYSIS DESIGN 

This report will identify and describe factors affecting innovative projects by 
means of multivariate statistical analyses of the survey data. We collected quantita- ^ 
tive evidence and used it to examine three broad, interrelated inquiries: 

• To what extent did differences between the federal change agent programs 
explain variations in the implementation and continuation of innovative 
projects? 

• Which characteristics of innovative projects significantly affected their 
' implementation and continuation? 

• Were differences in institutional settings related to variations in project 
implementation and continuation, controlling for other factors, and, if so, 
which institutional aspects had significant and important effects? 

" To place these major empirical efTorU within the context of federal anri .state policy, the following 
additional steps were taken by the Rand staff telephone interviews with 54 SKA oflicials in 18 states, 
visits to nine SEAs to interview various state officials, and a series of personal discussions and interviews 
with USOE, HEW, and Congressional staff concerning federal [wlicy issues m change agent programs. 
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Although the answers to these inquiries must necessarily be incomplete, tliey, along 
with the companion fieldwork results, provide systematic knowledge to help us 
address the major policy questions motivating the change agent study. 

We will outline below the survey and sampling procedures we designed to col- 
lect the data. The next section discusses operational measurements for various 
hypotheses implied by these questions. 

Survey and Sample Specifications 

Our basic research approach is deceptively simple. Using a personal interview 
survey of project participants in local school districts (backed by selective, intensive 
fieldwork), we collected data on the dependent variables— the implementation out- 
comes and expected continuation of individual innovative projects— and descriptive 
and attitudinal data on the factors or independent variables— project character- 
istics, institutional setting, and federal and state policy. The statistical analysis of 
this information consisted of determining the effect of any independent variable on 
implementation outcomes or continuation taking into account the other indepen- 
dent variables. 

This "simple" approach masks major obstacles that required complex research 
design and analyses. First, since systematic knowledge of the factors aff'ecting inno- 
vations and educational change is quite primitive, the survey design was comprehen- 
sive and exploratory rather than sharply focused. 

Second, the projects funded by the federal programs v*ere extraordinarily di- 
verse, ranging, for example, from complex performance contracting techniques to 
field trips to local cultural attractions, from open education to computer-assisted 
instruction, from remedial reading in elementary schools to integrated curriculum 
in high schools. Describing project characteristics in comparable and analytical 
ways thus became a major design and analysis consideration. Accordingly, the 
sampling procedure was designed to cover a wide range of innovative projects, and 
the survey was designed so that various project participants were asked many 
questions about the nature of their project. Statistical methods were then used to 
describe the range of project characteristics.^ 

The complexity of the organization of the school district also had to be inte- 
grated into the research design and analysis. Because we assume that decisions and 
processes influencing innovation are not confined to one central location, data were 
collected for different levels in the school organization. For example, we hypothe- 
sized that the superintendent and school district officials play major roles in the 
initiation and continuation stages of innovations, but often do not directly affect the 
implementation stage. Accordingly, the personal interviews with teachers, princi- 
pals, and project directors focused on questions of implementation, vvhereas those 
with superintendents and federal program managers dealt with initiation. 

» Projects whose intent was not related to innovations affecting chissroom activities were not included 
in this study-for example, Projects that relied solely on home visits or only involved curriculum deyelop- 
ment^he third section of App A det^uls the excluded categories. Although our fundamental interest 
is understanding innovations and what determines their outcomes, it is difficult even after considerable 
dat^ have been gathered to say whether a given project is "truly innovative. Attempting to choose a 
sample of innovations only on an a prion basis would have been a costly and uncertain undertaking 
Therefore, we limited the analysis to projects that were funded by the federal demonstration programs 
The sample consequently ranges from highly innovative projects to those that barely meet the loosest 
definition of an innovation— a project , that is new relative to a classroom, school, or district. 
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On the basis of these considerations, a series of personal interview schedules 
was de^^igned by Rand and administered by the National Opinion Research Center 
to a nationwide sample or293 projects in 18 states funded by the federal demonstra- 
tion programs The basic design involved the selection of 224 projects in 196 school 
districts in which the superintendent, federal program manager, principals from 
two project schools, and two teachers from each of the two schools were interviewed 
using a primarily close-ended questionnaire. Respondents were asked about: 

• The characteristics of their project (e.g., its resource level and target group 
size, substantive focus, educational techniques or methods, and the steps 
taken to implement the project). 

• The innovative process as it worked for their project (e.g., origin of the 
project and difliculties during implementation). 

• The institutional setting (e.g., the organizational climate in the schools; 
the support from principal actors; and the social, political, and economic 
characteristics of the district, the school, and the students). 

• The respondent's experiences with, and attitudes toward, federal and state 
programs and personnel, perceptions of project outcomes, and personal 
background characteristics. 

The first section of App. A describes the survey instruments in detail. In addition 
to the 224 projects, project directors from other projects in the preselected districts 
were interviewed whenever the opportunity arose (increasing the sample of projects 
by 69). 

The basic sampling procedure involved two stages: a sample of 18 states fol- 
lowed by the selection of federally funded projects within states. The purpose of the 
state sampling was to obtain a sample that was broadly representative of region, 
level of educational funding, and intensity of educational management at the state 
level. Subject to these criteria, the selection of states was weighted according to state 
population. Projects within the 18 states were selected to cover the range of diverse 
project characteristics and of institutional settings. That is, using information from 
IJSOE files on the nature of projects and from the census on the demographic 
characteristics of the school districts, samples were chosen that excluded irrelevant 
projects and that provided sufficient numbers of projects for critical school district 
attributes (e.g., size, urbanness, race and ethnicity, and socioeconomic status). 

Table 1 lists the number of interviews actually obtained for each of the principal 
types of respondents by each federal program. Table 2 shows the distribution of 
sample projects for each state by federal programs. The net result of the sampling 
procedure is not a probabilistically representative sample but a sample that covers 
a broad range of projects in a wide variety of local and state settings. As the last part 
of this section details, the purposive nature of this sample permits tho drawing of 
statistical inferences of the type necessary to investigate the study's central re- 
search questions. The second section of App. A details the sampling procedure and 
the nature of the sample obtained; the third section of App. A compares the sample 
obtained with the national school district population. 

Data Analysis Plan 

An explanatory model or series of assumptions and hypotheses about school 
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Table 1 

NUMBKR OF PKRSONAL [NTERVLKWS BY PROGRAMS 



Respondent 


Title III 


Right- 
To-Read 


VII 


Vocational 
Education 


Total 


State 


Federal 


State 


Federal 


Projects/project 
















directors 


176 


18 


32 




16 


10 


29 3 


Superintendents 


115 


11 


22 


24 


12 


10 


194 


Federal program 
















managers 


lU 


11 


22 


24 


12 


9 


191 


Principals 


213 


32 


36 


44 


23 


20 


368 


Teachers 


397 


58 


75 


83 


41 


35 


689 



Table 2 

NUMBFR OF SAMPLI- PROJECTS BY STAlii 



State 


Title 1 1 I 


Right- 
To-Read 


■i\itle 


Vocational 
Sducat ion 


Total 


State 


Federal 


Slate 


Federal 


Arkansas 


4 








1 


1 


6 


CaliTornia 


32 


4 


3 


<h 


1 




49 


Colorado 


5 






3 


1 




9 


Florida 


6 


2 


1 


2 




1 


12 


Georgia 


4 




2 




1 


1 


8 


Illinois 


27 


2 


2 


2 






33 


Kentucky 


7 








2 


1 


10 


Massachusel ts 


6 


1 


2 


4 






13 


Michigan 


6 






2 






8 

12 


Missour i 


6 


2 


2 






1 


New Jersey 


21 


1 


2 


2 


2 


1 


29 


New York 


19 


2 


a 


9 


1 




39 


North Carolina 


5 








1 


1 


7 


Ohio 


6 




2 




3 


1 


12 


Pennsylvania 


4 


i 








1 


6 


Texas 


4 


1 


6 


7 






18 


Washington 


7 


2 






2 


1 


12 


Wisconsin 


7 




2 


I 






10 


Total 


176 


18 


32 


41 


16 


10 


29 3 
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district decisionmaking forms the ioundation for the data analysis. The model, 
which vviM be discussed in Sec. 11. implies that two levels of analysis should be 
pursued— the school/classroom level and the school district level. At the school/ 
classroom level, implementation is the primary concern; at the school district, ex- 
pected continuation is appropriate for analysis. 

For each of these levels, the data analysis involved twostep^ First, independent 
and dependent variables, as appropriate for the central research q^uestions, were 
selected and measured. In particular, the independent variables measure important 
project characteri.stics, aspects of the institutional setting, and federal program 
inputs; the dependent variables measure implementation outcomes and expected 
continuation. 

Second, we used multiple regression techniques to estimate the effects of an 
independent variable on the outcome measures. This statistical approach allowed us 
to test the significance of the variables as well as suggesting the relative importance 
of the independent variables in explaining variations in project outcomes. 

Section II discusses the explanatory model and describes the characteristics of 
the independent and dependent variables relevant to implementation of the class- 
room level. Section III describes the management strate^'^es r''the federal change 
agent programs and examines the edects these programs had on the design of 
projects and on the difficulty of implementing them. Section IV estimates the effects 
of the independent variables on implementation outcomes at the school/classroom 
level; Sec, V does the same for expected continuation at the district level. 



SCOPE AND LIMITATIONS OF THE ANALYSIS 

Because this inquiry is' ambitious, its scope and limitations should be made 
explicit. Three types of considerations limit and qualify the study. First, being one 
in a series of volumes, this report deliberately focuses the discussion on quantitative 
evidence primarily from the survey Thus, we will not deal with the process of 
change itself— the subject of Vol. Ill— but only with the project characteristics, 
institutional settings, and federal and state programs that affect educational change 
and thereby the implementation and continuation of projects. Accordingly, we will 
not "flesh out'* the findings with the highly interesting and important case study 
material presented in Vol. III. Nor will we describe the federal demonstration 
programs except as necessary for quantitative analysis or inference The theoretical 
context of the entire study is treated in Vol. I, and we assume that the reader is 
familiar with that presentation. Finally, we do not reflect systematically on policy 
conclusions that might be drawn from the quantitative evidence. Such inferences 
are better made in conjunction with the parallel fieldvvork study. Volume IV synthe- 
sizes the quantitative findings presented in this report with the fieldwork findings 
presented in Vol. Ill and suggests some tentative policy implications from the first 
phase of the change agent study. 

Other limitations are intrinsic to the survey methodology, its design, the sam- 
ple, and to statistical analysis in general. The survey necessarily sacrificed in-depth 
questioning for cross-project comparability. A loss of detail and substance was thus 
incurred. Of course, details often are idiosyncratic. The survey's strength lay in 
gathering data relevant to major pattern.s, not in deciphering complex processes. 
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The survey was vulnerable to systematic biases as well as to random distortions 
on the part of the respondtMits. But so are most nonobservational data-gathering 
methods. Without in any way discounting the seriousness of this problem, it should 
be recognized as one of degree. Project directors probably tended to believe their 
project was more successful than an objective outsider might say. Moreover, they 
might have been even more optimistic when responding to a federal eva^uator than 
to a social scientist, but less optimistic when complaining to a spouse. Insofar as 
possible, we tried to reduce these measurement errors.by using multiple measure- 
ments, by checking the consistency of answers of the same respondent and the 
consistency of answers to the same questions for different respondents, and by 
checking answers to critical questions with a validation instrument designed to tap 
the assessments of Rand fieldworkers at the case study sites. Finally, we eliminated 
data from the analysis for which the distortions appeared too great. 

Nonetheless, response bias existed in the data. For example, the average per- 
centage of goals achieved on a project was reported by project directors as being 
approximately 75 percent. This average might have been **distorted" in two ways. 
First, it might have been systematically higher than an average from observations 
of disinterested parties. Second, the extent of distortion might ha^e depended on 
systematic land nonsystematic) characteristics of the project and the respondent. 
We believe that the first type of problem— being overly optimistic— was the major 
problem. Accordingly, we did not rely on the absolute level of such answers. How- 
ever, since our analysis deals esseniially with comparing projects, the perceived 
success of project A relative to project B (and what explains this relative success) was 
more pertinent than the **correct" absolute levels.*'^ 

We also avoided absolute assessments of success for another reason; The sam- 
ple was not statistically representative. For example, even if 75 percent average 
success of projects had no measurement error, 75 percent would not have provided 
an accurate estimate of the average success of all projects funded by the federal 
demonstration programs (nor would the average of projects within programs provide 
a reliable estimate of program success). However, the sample did allow us to make 
relative and comparative statements for many of the crucial concerns investigated. 

A third general qualificaticn of the results reported here falls under the head- 
ing of perspective. This study describes interim analyses of the first phase of a 
complex and ambitious two-phase undertaking. Whereas we are confident about all 
those statements we call conclusions, additional and more refined analyses will be 
undertaken to elaborate the findings and to hypothesize new ones. The results 
should, in short, be treated cautiously. 

'° Insjofar as distortions vary systematically with factors explauung effective implementiition or 
continuation, the analysis based on answers that are relatively accurate is subject to error We believe 
this problem to be less prevalent in the data chosen for anal>sis than the absolute bias mentioned above 
However, when we suspect these errors m any given analysis* hall indicate them We were ofUn able 
to avoid the problem ofcorrelated error terms* by acareful? ' -ction of u hoi?c responses were analyzed 



II. SELECTION AND MEASUREMENT OF VARIABLES 



This section outlines the exphinatory model on which the data analysis was 
based and describes tht- choice of independent and dependent variables used in the 
of classroom implementation outcomes presented in Sec. IV. 



DATA ANALYSIS MODEL 

The analysis of factors affecting the outcomes of innovative projects is predicat- 
ed on a hypothesis derived from our fieldwork experience: On one hand, school 
district officials generally make crucial decisions during the initiation and continua- 
tion stages of innovations rather than during implementation and, on the other 
hand, teachers, project directors, and principals are the primary actors during 
implementation. District officials are usually too busy and too remote to become 
involved in ihe day-to day implementation of projects. Teachers and other project 
participants seldom have the authority or influence to play direct roles in district 
Allocation decisions about the future of projects. Accordingly, the data analysis was 
divided into two levels, project implementation at the classroom level and project 
continuation at the district level. 

Figure 1 presents a schematic diagram of the data an.w^sis model that indicates 
the type of analysis planned within each level and the relationship between the two 
levels. For the classroom level, we shall consider implementation outcomes to de- 
pend on project characteristics, aspects of the institutional setting, and the federal 
change agent program funding the project. Both independent and dependent vari- 
ables consist of sever.-^i components that will be described in operational terms in 
succeeding sections. 

We assume that the decision to continue a project using district resources was 
essentially made by the superintendent. This decision was, we hypothesize, a func- 
tion of the perceived success of the project, project characteristics, political-economic 
factors, internal organizutional considerations, and federal change agent policy. 
Section V identifies these variables more specifically and describes theii operational 
measurements. 

Although the two levels of analysis are conceptually distinct, they are presumed 
to be linked in tv/o ways. First, the superintendent s perception of the success of a 
project is assumed to have been based on how the project was implemented. (The 
downward arrow in Fig. 1 represents this linkage.) That is, the superintendent's 
perception was presumably derived from information about project outcomes as 
they were communicated by project participants from the various schools imple- 
menting the project. Consequently, the statistical analysis presented in Sec. V con- 
sisted of a first stage that estimated the superintendent's perception of project 
success as a function of the attitudes of project participants and a second stage that 
used this estimated perception of success as one input to the continuation decision. 
The analysis will test whether perceived success was positively related to continua- 
tion of a project 

In 
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CLASSROOM LEVEL (Ser IV) 

Project Characteristics 
ImplementaliJti Strategy 
Resources 

Hlducational Technolog>^ or Method 

Institutional Setting 
Organizational Climate 
School/Classroom Characteristics 
Characteristics of Principal Actors 

Federal Programs 



DISTRICT LEVEL (Sec. V) 
Project Characteristics 
District Political-Economic Factors 
Organizational Decisionmaking 
Federal Programs 



Implementation Outcomes 

Percentage of Goals Achieved 
Teaclier Change 
Difficulty of Implementation 
Fidelity of Implementation 



Perceived Success 



Flxpected Continuation 



Fig. 1— Data analysis model 



A second link between the levels may occur in the relationship between organi- 
zational climate and implementation. Among the elements composing the organiza- 
tional climate or milieu in which the teacher must implement an innovation is the 
support or lack of support of the superintendent. We will test whether the superin- 
# tendent's support affected the success of the project's implementation. 



IMPLEMENTATION OUTCOMES 

Although student outcomes might be the ultimate indicator of the effectiveness 
of an innovation, it was both premature and inappropriate to attempt such measure- 
ment in this study. As we pointed out in Sec. I, the projects studi^ here were 
generally new to the district or school and usually needed time for development 
before anyone could make an accurate assessment of their long-run effects on stu- 
* dents. Moreover, the change agent projects were so diverse in their goals and ambi- 
tions that comparisons across projects on absolute or content-free student perfor- 
mance measures were meaningless. For example, an increase of a half grade level 
in cognitive test scores for a remedial mathematics project in an elementary school 
cannot be readily compared with a measure of greater awareness of career oppor- 
tunities for a high school career education project. 

Important conceptual reasons also lead us to measure project ''outcomes" other 
than student outcomes. Projects must go through the complex and uncertain process 
of implementation before they can affect students. It makes sense to put first things 
first and to measure the effectiveness of implementation before examining potential 
stua it impacts. Moreover, projects may "mutate'' during implementation. Unless 
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one can predict how the project changes^ it is not possible to determine whether 
variations in student outcomes are the result of the initial project plans or of the 
interplay of the project with its institutional setting. In short, for the purposes of 
this study, we focused on implementation ^ind developed measures of the effective- 
ness of a project's implementation. 

We employed four measures that tapped different yet related aspects of the 
efTectiveness of implementation. One aspect was the e.xtent to w'hich a project wao 
implemented as originally planned or, for short, the fidelity of the implementation. 
Our fieldwork experience suggests that some projects encountered so many prob- 
lems of one sort or another that they never ''got offthe ground" and thus were not 
implemented; they did not provide a true test of the educational treatment. Many 
of the innovations we studied were designed as demonstrations; therefore, the fideli- 
ty of implementation to the projects design is a useful indication of outcome and 
consequently was measured as a dependent variable. 

Even if a project were implemented as proposed, it might fail to reach desirable 
goals. The innovation might be no better than what it replaced. Thus, it is reasonable 
to measure the extent to which the project achieved desirable goals, that is, its 
perceived success. This outcome was measured independently of the fidelity of im- 
plementation in the sense that the original goals may or may not have been followed 
or evaluated. Indeed, we observed projects, particularly ambitious classroom organi- 
zation innovations, that altered or redefined their original goals during implementa- 
tion. Survey questions about perceived success were asked in the last year or years 
of the projects* federal funding. Because these change agent projects may not have 
been m operation a long time relative to the time required for some innovations to 
take root, perc^^ived success measures shorl-run achievement of goals. Thus, fidelity 
of implementation was distinguishable from perceived success, and different factors 
can be expected to have afTected these outcomes differently. 

A third possible result of a project's implementation was change in the behavior 
of teachers and administrators. A project might have been deliberately planned to 
introduce specific kinds of change, or the changes might have been an unanticipated 
consequence of implementation. In any event, it can be argued that unless teachers 
alter their tr^^ditional behavior, significant innovations cannot be accomplished and 
maintained. Projects that induced changes in teachers* behavior were not necessari- 
ly successful in te -ms of achieving desirable educational goals (e.g , some stafTdeve- 
lopment projects introduced new teaching styles or practices but did not have posi- 
tive effects on students because an extended period of implementation wa^ neces- 
sary). Therefore, we will test whether different independent variables were signifi- 
cj^ntly rel|\ted to changes in teachers' behavior .^nd will compare these effects with 
those for perceived success and fidelity of implementation: 

A final dimension of the outcomes of an innovation was the organizational co.^its 
or difflciiliy of implenientation. One objective of demonstration projects was to 
generate information for the ^chool, school district, and oth^er potential users. 
Knowledge about how hard it is to put^n innovation into practice is thus an 
important outcome of the implementatioa stage. Projects that^ere hard to imple- 
ment may or may not have remained, true to their original design, may or may not 
have resulted in significant change in teachers' behaviors, and may or may not have 
achieved desirable goals. ' • " ^ * ^ 

These four measures of irhplementation effectiveness or outcome allow us to 



ERLC 



15 



determine whether difTerent factors affect different aspects of implementation and, 
therefore, enable us to make overall judgments about implementation. For example, 
project characteiistics that are positively related to fidelity of implementation may 
be negatively related to teacher change. This result would be consistent with the 
following hypothesis: Unless the project deviates from its original proposal, that is, 
unless it adapts, teacher change may not occur. As another illustration, project 
characteristics that are positively associated with perceived success but negatively 
associated with teacher change might reflect (depending on the characteristics in 
question) "successful" but trivial projects. 

The survey asked several questions of each respondent type (teacher, principal, 
project director, federal program manager, and superintendent) about their percep- 
tion of the above outcomes. Although the use of these self-perceived measures in- 
troduces "subjective" biases, which we will discuss, they have the important advan- 
tage of pr6vidingdataon the participants' perception of the implementation of their 
project. These self-perceived assessments are thus automatically adjusted so that 
they are relative to the local context of the project. On the basis of a series of 
reliability and validity analyses, we selected the following operational measures. 

To measure perceived success, we employed a simple question that was intended 
to elicit the self-perception of teachers on the extent to which the goals of the project 
were achieved.* This question was subject to a variety of response biases. Partici- 
pants in projects tended to be more optimistic in their evaluation of success than 
were ou^sid^ observers.^ This systematic exaggeration was partly idiosyncratic or 
random and partly related to the role of the participant. Thus, project directors gave, 
on the average, the highest success rating, followed by the federal program manag- 
ers, superintendents, principals, and teachers. We resist the temptation to speculate 
on these discrepancies except to note that the perception of success (percentage of 
goals achieved) related to the respondent's own frame of reference and therefore was 
linked to his role. Whatever the reasons for these differing answers might have been, 
two conclusions for measurement follow. First, a measurement of success combining 
respondents' answers at different levels of the organization (e.g., superintendent and 
teachers) would have induced serious validity problems. Second, the teachers' an- 
swers were the ones closest to those of disinterested observers and, considering the 

' The specific question and statistics for the distribution of responses were: 
Overall when you take into account the goals \ou started with and the resources you had, about 
what percentage of the project*s goals would you say were achieved? 

Mean = 73.1% 
Median = 80.0% 

Standard deviation = 19.7% ^ 

* Number of responses = 642; no answers = 47. 

* Using the fieldwork sample for validation purposes (29 projects), the Rand observers rated their 
projects systematically lower than the reports of participants. However, this "absolute" exaggeration (or 
upwaid bias) did not seem to be greater for some types of projects than for others. It should be noted that 
the Rand fieldwork sample and thus the validation sample was biased toward more successful projects. 
Five alternative measures of success were used in the preliminary analysis Although all the measures 
were reasonably correlated (for the same respondent but not highly correlated across different respon- 
dents on the sa.iie project), the validation check and a series of internal consistency tests indicated that 
the success measure used in the test was the most valid, reliable, and fruitful. 
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teachers* involvement in the day-today operation of the project, the ones most likely 
to have reflected implementation at the classroom level.^ 

Although we thus have tried to guard against undue distortion, it would be 
naive to trust the absolute average figures from any project participants, even 
teachers. But it is the relative figure — comparing one project with another— that is 
particularly important in our central concern of determining the effects of various 
factors un implementation. Hence, the absolute distortions in our measure of pei 
ceived success can be tolerated. 

Aside from measurement errors, the interpretation of perceived success must 
be carefully understood. Percentage of goals achieved was a surrogate for the success 
of a project in achieving desirable goals, not a measure of the extent to which the 
project produced effects on students that were comparable across projects. This 
surrogate avoided the value problem of trying to weigh the importance of reading 
againsi mathematics -or enrichment. Project participants implicitly defined their 
own goals. 

In lieu 0^ direct observation and because of the variability in pruject character- 
istics, we used a neutral measure to quantify changes in teacher behavior. We asked 
teachers to judge how much^the project had caused them to change the way they 
did things in their classroom. Similar considerations of reliability..and validity 
apply to this measure of teacher change as those discussed for percentage of goals 
achieved. Consequently, we need not repeat that discussion. Nor need we dwell on 
the operational measurements of the other dependent variables, namely, the fidelity 
of implementation measured as the degree to which a project was implemented as 
initially laid out^ and the difficulty of implementation.® 

' It IS alto|ether"possible that participants on "unsuccessful" projects exaggerate their success more 
than those on successful projects. Unlike the distortion mentioned above, this "correlated error ' would 
seriously affect the statistical analysis. Howeva-, teachers have little to gain from distorting their 
answers m the context of this inten'iew situation. On the contrary, the main reason why teacher success 
reports exceeded those of the Rand observers may be that teachers lack Rand's overall perspective of 
having seen a wide range of projects, some of which were very successful. 

* The specific question and the frequency of response were: 

Overall, would you say the project has caused a great deal, some, very little, or no change in the 
way you actually do things in your classroom'^ 

Coded as 



^ Great deal 31.66% 4 

Some 43.89% 3 

Very little 15.02% 2 

None ^ 9.43% 1 

Number of responses = 674; no answers = 15. 



^ The specific survey question and frequency of response were: 

In general, to what extent icould you say the project at your school has been implemented as 
initialh laid out in the proposal? Would you say almost completely, somewhat, very little, or not 
at alU' 

Coded as 



Almost completely .50 27% 4 

Somewhat 37.77% 3 

Very little .5.17 % 2 

Not at all 0 78% 1 

Number of responses = 630, no answers = 59. 



* The specific survey questiqp'and frequency of response were: 



n 



These four measures— percentage of goals achieved, behavioral change in 
teachers, the degree of implementation as laid out, and the difficulty of implementa- 
tion — were systemically interrelated.^ However, because we wanted to balance the 
results for each measure against one another in order to arrive at an overall assess- 
ment, we analyzed each measure separately for the same factors and compared the 
results (see Sec. IV).® 



INDEPENDENT VARIABLES AT SCHOOL/CLASSROOM LEVEL 

Three classes of independent variables might have affected implementation 
outcomes— project characteristics, the institutional setting, and federal "program 
characteristics. We next describe measures of the first two classes. A discussion and 
analysis of the federal change agent programs as independent variables is deferred 
until the next section. 

Project Characteristics 

The literature on the evaluation of educational innovations usually deals with 
the technology (alias treatment, method, qr practice), the goals, and the resources 
of projects but largely ignores the project's implementation strategy— that is, the 
design choices made in order to implement the innovation. Yet implementation 
strategy may play the most important part in determining a project*s outcomes. The 



In your judfiment. how difficult has this project been for the teavhem here to carry out: very 
difficult, somewhat difficult, fairly easy, or very easy^ 

Coded as 

Very difficult 8.52% 4 

Somewhat difficult 42.14% 3 

Fairly ca<iy 37.59% . 2 

Very easy 11.75 % 1 

Number of rcsponsos - 676, no answers = 13 

' The pairwise correlations were not as high as one might expect Because of the low correlations, 
the procedure of analyzing these variables separately was justified. The pairwise correlations were 

Percentage Imj)lemen- 
of Goals Chanj^e tation 

Change .14 

Implementation 34 15 

Difficulty -.13 .15 -.08 

Since as many as four teachers were questioned about the same project, the above correlations across 
all teachers reflect withm-project teacher variation and thus might be lower than comparable project- 
level figures. Although such is the case, the following correlations for the same variables averaged across 
projects remain sufficiently low to justify the above approach- 
Percentage Implemen- 
of Goals Change tation 

Change 35 

Implementation .60 .24 

Difficulty -.17 .26 -.03 

* These variables affected each other simultaneously In lieu of a .mathematical model of these 
interrelationships, we estimated them separately in reduced form 
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analysis distinguished these different characteristics of projects, measured them, 
and tested differences in their effects on project outcomes. Given below are our 
measures of project resources, educational methods, and implementation strategy.® 

Project, Resources. The resources available for a project can be specified in 
a variety of ways. In our analysis we treated three dimensions of project sources— 
landing level, number of students in the project, and funding level per student in 
the proj<*ct 

Funding Level and Project Grant Size. Although federal grants in aid were 
generally designed to bear the greatest share of the increased costs associated with 
an innovative Project, they were not the only source of project funds. Many change 
agent projects obtained substantial funds, in addition to the federal change agent 
project grant, from state categorical.programs, special LEA appropriations, or pri- 
vate foundations, Moreover, some LEAs had "packaged" federal grants from such 
sources as Title I, EPDA,*° or NDEA** to supplement the primary change agent 
grant. For the evaluation of the effect of incremental resources on project outcomes, 
we used information on the sum of special grants from all sources that supported the 
innovative project. 

However, many projects in the Rand sample experienced considerable variabili- 
ty in funding over the several years they existed. What was the best summary of the 
'Tiscal scope" of a project that received $40,000 in its first year, including a $10,000 
plan ning grant; $125,000 in its second year; and $75,000 in its third year? The simple 
measure of average grant received over several years could be deceptive because of 
year-to-year variation; for example, a project primarily concerned with introducing 
new technology and hardware to a school may have a large capital grant in one year 
and much smaller support and maintenance grants in othar years. Various analyses 
indicated that the most useful measure for present purposes was the-largest yearly 
funding level, that is, the sum of special project grants in the year of greatest 
expenditure. 

The frequency distribution of the largest yearly funding level of projects in the 
Rand sample is shown in Fig. 2. Although several change agent projects depended 
on federal grants that in any one year never exceeded ?75,000 (about 90 of them), 
most projects had at least $100,000 in grant funds available for implementing an 
innovative project.*^ The distribution of totaKgrant packages is highly skewed; that 
is, many projects had funding levels less than $150,000 and fewer projects had very 
large grants. 

Number of Project Students and Funding per Project Student. The range 
of target group sizes for change agent projects in the Rand sample was large: From 

* We could not measure project goals in comparable ways. Altliough a variety of questions was asked 
of various types of respondents about the specinc nature of a project's goals, there was little comparability 
in their answers. This may reflect either a poor choice of questions or a lack of specificity and clarity 
intrinsic to educational innovations. In either event, the most useful coding was on the basis of the 
substantive focus of the project. For the Right-To-Read, Bilingual, and Career Education projectB the 
substantive focus is obvious For Title HI projects, it was possible to distinguish among reading, bilingual, 
and career education projects. Subsequent analysis employs some of these categorizations. 

Education Professions Development Act 

National Defense Education Act. 

Ao previously noted, the various federal programs differ in their funding levels and size of grants. 
An exclusion criterion in the Rand sampling procedure was a cutoff' level of $10,000 for any project. 
Thus, the Rand sample neither includes nor represents the small projects of $10,000 per year or below 
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Fig. 2— Largest yearly funding level (28 projects over $300,000 not shown) 



fewer than 50 to more than 5000 students may have benefited from a single project. 
The frequency distribution of the Size of projects in the Rand sample is given in Fig. 
3. Much of the variation in project size is associated with school district size (total 
enrollment), and consequently with urbanness. But there were often small projects 
within large school districts. 

By combining the information on the size of change agent grants and on the 
number of students served by the projects, we derived a measure of the concentration 
of project resources. The project grant per pupil in the target group (in the project's 
biggest funding year) can be taken to indicate several phenomena: 

• The expense per pupil of introducing and maintaining a project. 

• The degree of focus on particular students (or dispersion among students) 
in a project. 

• The policy intentions and priorities of the funding source. 

Table 3 shows the pattern of concentrating resources for each of the federal grant- 
in-aid programs. Only the State Title III program had enough observations to permit 
good comparisons, but we may note that it supported more projects that spent less 
than $25 per student (24.8 percent of State Title III projects) than did the other 
federal programs (11.9 percent of the five oth6r program projects). Still, almost a fifth 
- of the Title III projects spent more than $450 per target student. 

In sum, there was clearly variation in the sample for the three measures of 
project resources and concentration: the absolute level of project funding in its 
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Percentage ot Sample Projects Funded 
by Federal Program 
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biggest year, the size of the target group, and the funding level per project student. ^ 
Subsequent analysis will investigate whether the differences in resource^ affected 
project outcomes, all other things being equal. 

Educational Methods. The designers of projects at the level of the local 
school district may not, and often do not, plan in neat analytical terms that separate 
means from ends. Rather, they begin with a set of ideas consisting of an intermixture 
of goals, techniques, and strategies. Nonetheless, for comparative analysis we can 
separate the educational technology from other aspects of project design. We next 
suggest an analjrtical classification of the educational method or techniques of the 
innovations in the Rand sample. 

The survey asked project directors to check off the educational techniques em- 
ployed in their project. Table 4 enumerates the educational techniques and lists the 
number of times each technique was mentioned. Considering the lack of clarity and 
difTuseness of educational treatments, it is not surprising that project directors 
never mentioned only one technique but indicated that their projects employed 
several techniques in combination. Thus, it would be confusing and almost meaning- 
less to identify a project as a "needs assessment project" or a **parent involvement 
project." Rather, it is more accurate to ask how extensively needs assessment was 
used, for example, and what combination of techniques was used in conjunction with 
needs assessment. The data, when analyzed by correlations among the various 
techniques, clearly suggest that certain techniques went together in practice. More- 
over, it seems reasonable to hypothesize that likely combine, ions of techniques were 
united by underlying and more general educational methods. Since there were 
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Table 4 

EDUCATIONAL METHODS OF PROJECTS 



Specrlal projects can call fov a varietij of 

afferent techniques. Please tell me which Number 

of the techniques on this card the project of Times 

makes use of, COPE ALL hEtmOllKl). Mentioned 



Needs assessment 224 

Paraprofessional staff (teacher aides, etc.) 188 

Instructional specialists 180 

Counseling and guidance specialists 90 

Performance incentives for students . . 100 

Educational technology (audiovisual materials, 

computers, etc.) 186 

Development of new curricula or materials 225 

Adoption of new curricula 113 

Field trips lAl 

Open classrooms 88 

Nongittded or ungraded classrooms 81 

Leaxming centers 1^9 

Peer instruction 

Team teaching 1^8 

Individualized instruction 234 

r-rhavioral objectives 216 

Diagnostic/prescriptive methods 181 

New management techniques (decision-assisting technology 

- such as PPBS,a MBO,b etc.) 81 

Parent involvement 201 

Use of community resources 185 



NOTE: Total respondents » 289. 
^lanning-Programming-Budgeting System. 
^Management by objective. 



relatively few likely combinations of techniques, we suspect that these general 
educational methods were also few in number. 

These theoretical premises suggest the use of factor analysis, a statistical proce- 
dure designed to reduce a large number of elements, such as diverse educational 
techniques, into a smaller number of underlying dimensions. The validity of factor 
analysis, apart from technical considerations, rests on whether the factors, or under- 
lying dimensions, have any valid interpretation or meaning. Our factor analysis of 
the data of Table 4 indicates that the interpretations of the factors were meaningful 
and provided an efficient means for analytically describing the educational methods 
of innovative projects in the Rand sample. 

Table 5 displays the results of a factor analysis designed to discriminate a small 
number of underlying or general educational methods. Five analytical types of 
educational methods emerged: 

• Behavioral modification instruments, involving techniques and instru- 
This report does not discuss the statistical details of the factor analysis. 
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FAC'IOR ANAIYSIS OF M)i;CATiON'Al METHODS 



Kdijcational TerhniqiK'S 
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.071 
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.57S I 



1^1 
1^» * 

US > 



138 



33" 



.033 
. 10^* 

.UA2 

.260 
. 



, 1"0 



.2:7 



.463 I 



,616 I 



257 



2 . 080 



School 
Administrative 
Changes 
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.019 
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. 32 1 



.5A3 I 
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.238 
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185' 
. 1.'.6 
.0b2 
.097 
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.650 I 



.'01 
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.392 

.A56 
.570 

.A97 
.A7C 
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.489 
.541 
.599 

.678 
.458 
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.629 
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menii designed to alter student learning behavior such as student perfor- 
mance incentives and various technological innovations. 

. Enrichment techniques, involving heavy components of field trips and 

' community resources and clearly not involving diagnostic methods. 

. Classroom organization methods, typified by open classrooms, nongraded 
or ungraded classrooms, and team teaching. 

• Intensive staffing of traditional teaching approaches. 

. Administrative changes in the school, such as new management tech- 
niques or the adoption of new curricula. 

The names of these underlying methods, which were chosen to reflect the likely 
, combinations of techniques revealed in Table 5, suggest general methods widely 
recognized by educational specialists. The prima facie intuitiveness of these general 
methods gives us reasonable confidence in their validity.^'* 

The factors, or underlying educational methods, may be interpreted in several 
ways. Each project in the sample received a score on each factor. A factor score can 
be thought of as representing the amount of a particular educational method for a 
project. For example, a score of 2.0 on the classroom organization factor means that 
an innovative project employed a very high degree of classroom organization change 
activities in its design.*' If an innovation had a high degree of classroom organi- 
zation methods and scored low on the other factors, we would be justified jn calling 
the project a classroom organization innovation.^^ Thus, any project in the Rand 
sample can be described in terms of its mix of the five educational methods. 

Subsequent analysis makes use of the factor score description of educational 
methods. As a preview of that analysis and a significant illustration of how the 
factors can clarify the issues at hand. Table 6 presents a comparison of federal 
change agent programs in terms of the average level of the various methods. The 
average values of the factor scores suggest that projects funded by Vocational Educa- 
tion primarily deal with enrichment methods; that Right-To-Read projects relied on 
intensive staffing using traditional methods and behavioral modification techniques; 
that Bilingual, projects did not involve school administrative changes but concen- 
trated on a combination of intensive staffing, enrichment, and some classroom 
organization changes. These quantitative relationships are in accord with the 
findings of the case studies on the federal programs as reported in Vol. III. 

Although both the Title III programs administered by the states and by the 
federal government allow projects to be formulated by individual LEAs for a broad 
range of objectives. Federal Title III projects were more likely to involve large-scale 
administrative changes in school organization (see Table 6)* such as the introduction 
of planning, programming, and budgeting principles. The Federal Title III program 
was announced in 1970 and subsequent yea: s in letters from USOE to each superin- 
tendent that indicated the approximate grant size (for 1971 projects, the announced 
USOE abjective was for grants averaging $150,000 for each of three years). These 

The vahdity of factor analysis also deFXinds on technical considerations not discussed here In 
general, the factor analysis of Table 5 has relatively good but not very strong statistical characteristics 
Other st^itistical methods using nonparametnc assumptions and clustering analysis yielded essentially 
similar factors. 

Factor scores are distributed as a standard normal distribution A score of 2,0 represents two 
standard deviations greater than mean project score 

The high-scoring proj' ♦s on the various factors were checked against fieldwork and other qualita- 
tive information and founu a) be pleasingly accurate. 
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guidelines may have encouraged projects of considerable scope, as well as projects 
favored by the superintendent's office. The resulting "top-down" style of innovation 
may account for the relatively high "school admini!=lrative change" factor score 
(.275) for Federal Title III projects. Conversely, the smaller gram size and greater 
spreading of resources among school districts that characterized State Title III 
projects allowed for the funding of more "bottom-up" projects of the type initiated 
by teachers in a single school. Therefore, there was less reason to expect State Title 
Hi projects to focus on organization-type changes. This contrast between the state 
and federal administration of Title III illustrates an effect of different guidelines and 
management strategies on project design choices. The next section will consider 
such effects in detail. 

Implementation Strategies. The design of innovative projects implies explic- 
itly or '"mplicitly a strategy for implementing the change anticipated by the project's 
goals and educational methods. That is, decisions are made— by superintendents, 
federal program , project directors, principals, and occasionally teachers— about 
planning, training, which schools to place the project in, etc. Such strategic choices 
largely determine how t^e inno.vations are implemented. In what follows we charac- 
terize elements of implementation strategies employed by the change agent projects 

The su:-vey instruments used in this study collected data about those various 
' elemf of implementation or planned change strategies identified in the literature 
as being effective in aiding the implementation of innovations in the schooi environ- 
mepc. In particular, the following strategic choices were measured:" 

* Planning 

<. Staff" training and development ■ 

♦ Project meetings 

_ < Actor participation 

>" An addiUona! impleme'ntatien strategy might.involve ai)proaches for introducing Ine educational 
jnalhod to the student. Such student strategies were seldom observed in the fielowork. and no systematic 
(fe^ were collected by the survey in this area. 
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• Implementation flexibility 

• Incentives offered to teachers 

• Change or effort required of teachers 

• Selection of schools and teachers 

• Concentration and location 

Section IV' will examine each of these elements to determine their influence on 
implementation outcome'^' For example, statistically controlling for other relevant 
variables, we will estimate the extent to which plannin,^ affected the perceived 
success of projects. 

The strategic components listed above could be defined in various ways. The 
exploratory nature of the survey meant that many of these elements were measured 
in broad and simple terms rather than in the in-depth, multiple-scale approach of 
more focused work. We believe that the survey s comprehensiveness in this unchart- 
ed area of implementation strategy compensates f ^ consequent loss in internal 
validity. Preliminary data work analyzed a large i iber of variables testing for 
reliability, validity, and independence from (or low correlation with) other explana- 
tory factors. This procedure reduced the number of implementation strategies used 
in the analysis presented in Sees. Ill and IV to the operational measurements listed 
by Table 7. To afford the reader a sense of the preliminary data> Table 7 also 
indicates items that will not be used in the analysis. 

Institutional Setting at School/Classroom Level 

We observed in the field a variety of idiosyncratic circumstances or events that 
affected the implementation of projects For example, personality conflicts between 
proje:t participants disrupted one project, and a change in overall management 
altered the course of another project. As important as such idiosyncratic occurrences 
may be. the quantitative analysis must focus on systematic aspects of the institution- 
al setting. In particular, three aspects that may have affected implementation out- 
comes were: 

• Organizational climate 

• School/classroom demography 

• Characteristics of principal actors 

Organizational Climate. We believe that the organizational climate of a 
school— thai is, the nature of its authority patterns, social and interpersonal rela- 
tionships, and esprit de corps— may significantly affect implementation. Given the 
exploratory nature of this study, we did not develop precise survey instruments that 
could be used for refined measurement of many important components of organiza- 
tional climate fe.g.» the principal s leadership style). Our primary concern was to 
provide general data to test the significance of organizational climate in broad 
terms. 

In particular, three aspects that could be tapped in a reasonably reliable and 
valid way were support for the project from various principal actors, the teacher s 

"* The followtng question asked of project directors will be used in subsequent analysis 

Ahmt how much support do you feel you haie reieived m your uork on this project fjom the 
district superintendent, the fedend pro^nwi nxdnager, pwjat prinnixds. the projei t faculty, and 
funding agency fKrsonnel: a lot, some, a hi tie. or mme^ 
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Table 7 

0PF.R.\TI0NAI. MKASURRMENT OF IMiM.l^'lENTATION STRATEGIES 



(1) 



Dei5criptlon of Variables Used 
In Preliminary Analysis 



Planning: Number of months spent In planning 
Tnd^wrltlng the original application; ivV- 

spent in I lannlnj r^rui I'VO^ect Usijn; 

th<i rt^thodo or natevUiU trw 

re 

S^aff trainjn^: Percentage of the project 
^budVet 'il located to training; the propor- 
tion of the teachers on a project receiv- 
ing training; W a^^ount of tipe (in :yc>c.> 
i.ivt icif-atirij tcaoht^vj si eni ik tiwlniy.j 
Project meetings. The frequency with which 
prl'nc Tpals' met with project staff; the 
frequency of project mee t Ings ; t^it? 

Actor_£artJ^cl2aULcm. The extent to which 
U^aahcrs and principals participated In 
various project decisions 

lm|>^l^mcmtatXonXlejia),Ul^ The extent to 
which tViOhero, principals, and project 
directors had freedom to alter project 
design characteristics during Implemen- 
tat Ion 

Incentives offered teacliers: Whcthe^V 
Chan^ or ef for^_re<l5iL^^ i>A Ji^iiSA^/^ • 



So 1 ec turn o t sc hoo 1 and _teache r s : How 
schools were' Vhosenj how teachers were 
chosen; the proportion of ceachers 
volunteoiing for the project;^ ni<rc^r 
jf ^tudefit$ :n the c:^i J^'^zc^ li ; < 
.V''it<*jV and Title 1 In 

the project school 

Concejurat Ion and location, ^h. r t^.r 
' fr -J. ^'c t ^^5(:r> ^- 1 ene n t arj s cjiL o I '^i Ij, 

■Ik ill jradLji the proportion of 
teachers at the school Involved In a 
project; whether the project was located 
solely in a classroom, iy: ^ si . '-nil am'. 
or outside of the school 



(2) 



Nam*>;j of Variables 
Used in Analysts 



Percentage of planning 

Materials not 

developed local ly 



Staff training time 



Value oi meetings 



Teacher not partici- 
pating In implement' 
at Ion dec Is Ions 

Teacher free to alter 
project design 



Teacher paid 

Overall teaching 
change required 

Change in bpecific 
teaching technique 
r equ i red 

Extra effort jequlred 
(principal 'i' view) 

Number of students in 

school 
Percentage of black 

students in school 

Percentage of Spanish 
students In school 

Pr .»ject covers high 
and Aienentary 
s< hool s 

Project located in 
spec lal unl t 



(3) 



Range of 
Var lables 



0-80 
0-1 



0-7 

0-<!. 

0-1 
0-1 

0-1 
0-1 

0- 1 

1- 4 

J5-36S3 
0-100 
O-lOO 
j-1 

0-1 



Respondent 
Used as Source 
of Variable 



Project director 
Pro ject di rector 



Pr indpal 

Teacher 

Project, director 
Teache r 

Project dirtn tor 
Pr inc Ipal 
Pr inc ipal 

Pr inc Ipal 
Pr inc ipal 
Pr inc ipal 
Pr ir.c Ipal 
Piojei t dire< tor 

I'roject director 



'^The Italics In the variable descriptions indi<.«t. 
naraos are given in col. 2. 



variables used m tht 'Inal ^nalv^ is vh-se 
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feeling of the school's morale,'*-* and the teachers extra effort expended on the 
project ''^ During preliminary analyses, each of these independent variables was 
checked against the answers of different rj.=;pondents for the same (or similar) ques^ 
tions and against the answers of the same respondent on related questions. These 
reliability and validity analyses will not be reproduced here. 

School/Classroom Characteristics. In addition to the internal organization- 
al climate of the school, a wide variety of school and classroom characteristics may 
systematically affect implementation--for example, the size of the school, the racial 
and socioeconomic composition of stud, nts in the school and classroom, and the 
innate attributes of students. This list could be easily extended. However, given the 
lack of a theoretical understanding in the literature about the way such variables 
might relate to project outcomes, we elected not to attempt major data collection in 
thes^ areas Rather, we gathered information on a small number of variables that 
could be used as basic controls for the multivariate statistical analysis. After 
preliminary data analysis, the following variables were selected for controls:^* 

1 Elementary schooi: coded 1 fur elementary school, 0 otherwise; number of 
elementary schools is 212. 

2 Number ofstudents in school; mean -738, median =588, standard devia- 
tion :^558. 

3 Percentage of black students in school: mean :^23, median ::;9, standard 
deviation —30. 

4 Percentage of Spanish-speaking students in school: mean =12, median - 
Ir standard deviation :^23. 

Characteristics of Principal ActoVs. 1* is quite plausible that personal attri- 
butes of individual teachers, principals, or project leaders, rather than their roles 

The specific question asked of teachers was 

On the whole, nould you say that the morale of teachers at this school us very hwh moderately 
htfih, o' not hi^h at all'' ' 

Coded as 

Very hfgh . 41.1.J% 3 

Moderately hi^h 'J9.3-i% 2 

Sot hif^h at all 9.52 1 

iCamber of responses = 684 

The specific question asked of teachers was 

About how much extra effort would you say r/iis project requires from project teachers now a 
lot. some, a little, or none^ 

Coded as 

lot 49,78% 4 

Some 37.44 % 3 

littli' > . 8.96 7r 2 

\'oric 3.82% 1 

Number of responses = 676. 

^' Data were collected on approximately twenty cnaractenstics. However, problems of multicol- 
iinearjty hmit the usefulness oremploying all cha^-actenstics as controls For example, we asked prmcj- 
pals to estimate the pro^wrtion of student^i m the school that were disadvantaged according to Title I 
definitions This variable is not used m the subsequent analysis because of its high correlation with the 
percentage of black students and the |>ercentage of Spa.-nsh.speaking students m a school 
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or positions, have major and pervasive effects on implementation. Yet e mating 
these effects requires measurement and sampling conditions that are .ficult to 
meet." Because of our other comprehensive concerns, the sampling id survey 
instruments of this study were not designed to test the importanf jf personal 
characteristics. Nonetheless, background data of the usual type-r . „ experience, 
race, ethnicity, and sex— were collected nd used in the preliminary analysis. The 
only'attribute that made a difference under these conditions was teacher experience 
(the number of years teaching) and, hence, it will be used in subsequent analysis. 



EXPECTED EXPLANATORY POWER 

The preceding discussion indicates a number of measurement and specification 
difficulties. These problems, which are to be expected in an exploratory study, are 
within tolerable limits. Perhaps the most serious deficiency of the data is our inabili- 
ty to measure and control for many persona! characteristics of principal actors, 
especially teachers and students. We suspect that a relatively large proportion of the 
variation in implementation outcomes may be related to the variability in personal 
characteristics that we cannot introduce mto the analysis. However, the findings of 
interest here may not be very sensitive to these o.aissions (i.e., specification errors). 
In any event, we do not expect the total amount of variation explained in implemen- 
tation outcomes to be high. Since our primary concern involves testing the signifi- 
cance of individual variables and hypotheses, a low overall percentage of explamed 
variation would not seriously hamper the analysis. 

" Attributes such as experience are relatively easy to mea.sure, whereas measuring whether a teach- 
er or principal has an open or closed personality is difficult. Moreover, the sample and the sUtist'cal 
model must be devised so as to distinguish among various elTects For example, to determine whether an 
S « the ^uU of a teacher's attributes or of the school that «.lected the teacher, it is necessary to 
have a sample that allows for analysis of both within-school variation and between-schoo! variation of 
teacher attributes 
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III. FEDERAL PROGRAM COMPARISONS: DESIGN 
CHOICES AND DIFFICULTIES 



The Rand study examined four federal change agent programs that had diverse 
management strategies and substantive priorities. This diversity provided an oppor- 
tunity to compare innovations funded by different programs and thus assess the 
extent to which variations in implementation outcomes could be explained by differ- 
ences in federal program strategies and priorities. In particular, this study will 
examine the extent to which the federal programs had differential impacts on: 

• Project design choices daring the initiation phase 

• Difficulties arising during implementation 

• Implementation outcomes 

• Continuation 

Analysis of the last two categories involves the comparison of the mean outcomes 
of projects funded by each federal change agent program with the mean project 
outcomes of other programs. This type of analysis can test whether, for example, the 
mean perceived success of Title III projects was greater than the mean of Title VII 
projects. However, to avoid specious inference, comparisons of this nature should 
take into account variations in project characteristics and in the institutional set- 
ting. Section IV will compare the federal programs with respect to implementation 
outcomes, and Sec. V will make similar comparisons for expected continuation. This 
section examines the extent to which project design choices and difficulties of im- 
plementation were related to particular federal change agent programs. To provide 
the reader with background material for these analyses, we first sketch the relevant 
elements of the programs. 



CHANGE AGENT PROGRAM GUIDELINES AND 
MANAGEMENT STRATEGIES 

Title III of ESEA 

Title III was the first major USOE program to provide funds to local school 
districts for innovation. Since the authorizing legislation places no restrictions on 
educational areas in which projects can be funded, Title III provides a broad pro- 
gram of support for local innovation. 

The goals of Title III are to stimulate and assist in the development of model 
elementary and secondary school programs through grants to local districts, and to 
support the spread of these models to other schools. Grants are awarded on the basis 
of competitive proposals submitted by LEA^. Title III is also intended to give school 
districts experience in managing in novation and encourage them to undertake local- 
ly funded efforts to innovate. 

When first authorized in April 1965, Title III was a Commissioner's program; 
funds were managed by USOE and went directly from USOE to local districts. Two 
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years later, Congress amended Title III to give the s^^tes responsibility for manage- 
ment of Title III funds. 

The State Plan Program. The 1967 amendments to Title III underwrote a 
great expansion in SEA staff and in the Title III program as a whole. SEA staff 
working on Title III have increased from the 50 full- and part-time persons working 
in 1967 to over 450 full-time and 483 part-time people today. 

The 1967 amendments also required that states set forth a plajitq be reviewed 
and approved by the Federal Title III officials for spending their allotment of Title 
III funds. The federal office was empowered to withhold up to 50 percent of a state's 
allotment until a satisfactory plan was submitted. 

The first official guidelines for the State Plan program were issued in the fall 
of 1971. These guidelines specified criteria for the states to follow in designing a Title 
III management plan that would be evaluated by the federal State Plans Branch. 
The guidelines outline seven main components for which the SEA was responsible: 

• Educational needs assessment 
. Project development 

. Selection and funding of model projects 
. Project and program evaluation 

• Validation 

• Dissemination 

• Adoption 

The State Plan strategy also included criteria for organizing Title III at the 
state level, notably the creation of State Advisory Councils and provision for profes- 
sional staff development. 

Section 306. In 1970-1971, three years afler the conversion of Title III to the 
State Plan program, Congress re-established a Commissioner's program in Title III 
by amending the legislation to establish the Section 306 program. Beginning in FY 
1971, the authorization provided that at least 15 percent of Title III funds be allotted 
USOE, with the remaining 85 percent allocated to the states. The Section 306 
program has the same broad legislative intent as the original Title III program and 
has been similarly managed. 

Budget. Congress has always appropriated more money for Title III than for 
other change agent programs. Local projects are typically funded for three years. 
Title III appropriations have been about $150 million a year, which is about twice 
the current budget for the largest of the other change agent programs (Title VII, 
Bilingual program). The appropriations for the Title III budget reached a peak of 
almost $190 million in 1968, after a rapid increase from $75 million in the first year. 
After 1968, Title III appropriations fell back and stabilized at a level of about $150 
million until FY 1975, when they were reduced to $120 million. The current appro- 
priation is about one-third of th authorized level. The Title III legislation allocates 
the funds among states by formula, specifying that states should receive $200,000, 
plus an amount in proportion to the school-age and total population of the state. 

■ U S Congress, PL 89-10. Sec 302(a) (2) /» ^ 
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Title VII, Bilingual Education Program 

The Bilingual Education program was enacted in 1968 as an amendment to 
ESEA The legislation recognizes ''the special educational needs of the large num- 
bers of children of limited English-speaking ability in the United States," and au- 
thorizes bilingual projects in local districts to meet these needs. More than any other 
program in the change agent study, the Bilingual program aims at public acceptance 
of a fundamentally new concept in public education; as a res ilt, it has been marked 
by turbulence and conflict. 

Guidelines. USOE spent the first five years (1968-1973) formulating the social 
role and objectives of bilingual education, developing curricula for different lan- 
guages and grade levels, providing in-service training for teachers, devising new 
assessment techniques, and stimulating the SEAs* interest by involving them in 
program management. Of all the programs in the change agent study. Title VII 
began with the fewest available resources and the least developed program strategy. 

Title VII focuses on poor children between the ages of 3 and 18 and areas that 
need bilingual education programs. The law, however, is vague about what educa- 
tional programs it envisions. In the first year of the Bilingual program, the federal 
program office sent grant announcements and guidelines to SEAs, which then for- 
warded them to districts that might qualify. Great leeway was given to LEAs so that 
they could focus on their own specific needs. LEAs could propose almost any educa- 
tion project for the target population as long as two languages were used in instruc- 
tion and the history and culture of the non-English language group were taught. 

The legislative language merely said that "Title VII funds are available for 
exemplary pilot or demonstration projects." There was little in the early guidelines 
to indicate that applicants would be participating in a nationwide effort to develop 
models of bilingual education. In later 3'ears, four components— instructional pro- 
gram, curriculum, community involvement, and staff* development— were urged on 
projects. But the initial guidelines did not clearly focus or structure the program. 
Approximately half of the bilingual projects in the Rand sample were funded under 
the 1968 guidelines. 

More recently, the Bilingual program office has developed a program manual 
that clarifies the definition of a bilingual education project and provides informal 
funding guidelines. After the first year, project and budget size could grow, but were 
limited to a vertical expansion in grade levels. For instance, if a program began with 
a kindergarten class in the first year, it could add a 1st grade class the second year, 
to allow for the development of a continuous program for the students. However, 
federal funds could not be used to expand the project horizontally, for instance, to 
two or more kindergarten classes in the second year. After the third year of vertical 
expansion, LEAs were expected to absorb the cost of the highest grade level par- 
ticipating. 

State Participation. The states had no official role in managing Title VII 
during the period covered by this report. SEA involvement with Title VII projects 
varies directly with state interest in bilingual education. In states that have not 
needed to focus on bilingual education, responsibility for Title VII programs is 
usually housed in some other bureau, such as Title I, migrant workers, foreign 
language coordinator, and so on. In states where bilingual education has been a 
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major concern, a fully functioning Title VII staff is the rule.^ These staffs have no 
formal direct authority over the funded LEAs, but they do exercise influence over 
Title VII programs by means of their connection with LEA officials and USOE staff. 
In western states, for example, Title VII is a higher priority for SEAs. Accordingly, 
they typically devote considerable staff time to monitoring projects, making regu- 
lar site visits, holding workshops, and the like. 

Budget. The budget of the Bilingual Education program has risen steadily 
ince its inception to its present level of $85 million in FY 1975, but did so only after 
a friendly Congress overruled administration requests to limit the program's 
growth. In the first year of the program, FY 1969, the program funded 76 projects 
in 21 states, with the $7,5 million appropriation. Most of the projects were in 
California and Texas, and by far the largest number included Spanish-speaking 
children. Most projects had budgets exceeding $100,000 and, following the legisla- 
tion, were sponsored by a LEA, a group of LEAs, or a LEA in conjunction with a 
college or university. The regulations limited projects to a maximum of five years 
of funding. 

After 1969, the proportion of grants to California and Texas dropped dramati- 
cally as the program grew and tried to spread to new areas. Legislation and program 
policy have encouraged more language groups to participate in the projects. 



Vocational Education, Part D 

The Vocational Education, Part D, program was enacted in 1968 as part of the 
comprehensive amendments to the Vocational Education Act of 1963. The goal of 
the Part D program is "to stimulate new ways of bridging the gap between school 
and earning a living for young people*' through providing grants to local districts 
for exemplary career education projects. Each project was expected to develop a plan 
that incorporated four elements: 

1. Broad occupational orientation in elementary and secondary schools to 
increase student awareness of the range of career options open to them. 

2. Work experience and cooperative educational studies. 

3. Specific training in job entry skills. 

4. Intensive occupational guidance and counseling during the last years of 
school. 

These goals are quite similar to what was later embodie'^ in the concept of career 
education. 

State Participation. Half of the Part D program funds ai^e managed by the 
states and half by USOE, The federal.program office wanted the federally managed 
projects to provide examples for state and local educators, to emulate or modify, 
using state or local funds. 

The federal portion of the Part D program encouraged proposals by an an- 
nouncement sent to the executive officers of the state boards for vocational educa- 
tion and the state directors of vocational education. They, in turn, publicized the 



* The information about the operation of SEAs was derived from informal interviews with state 
officials. ^ 

10 . 



34 



program and, solicited proposals from local educational groups. On request, the 
federal program office sent a prospective applicant a manual on how to prepare a 
proposal, the program r^ulations, a booklet on exemplary vocational education 
programs, and a brief bibliography of previous research put out by the Educational 
Resources Information Center (ERIC) The manual and regulations were purposely 
vague to allow local groups to formulate their own ideas within the parameters set 
up by the policy paper. No planning grants were given, and applicants were required 
to develop complete 36-month operating plans as their proposals.^ DHEW Secretary 
Finch decided to bypass this process in the case of 20 projects for Model Cities 
programs in specified cities. In these 20 cases, the Model Cities program planners 
were essentially given a grant and asked to write an operational plan for it. 

Beyond requiring that SEAs use funds to support exemplary projects within the 
categorical purposes of the legislation. Part D guidelines do not dictate state man- 
agement strategies, and the federal program office has no supervisory power over 
the SEA. Thus, SEA administration of Part D funds varied greatly, For example, 
SEAs had various strategies for selecting and funding their projects, A. number of 
states funded the same projects selected by the federal program office. Some states 
elected to fund their own large projects, comparable in size to the federal Part D 
project. Other states funded three or four smaller projects. One state funded hun- 
dreds of mini-grants. 

In states where a number of proposals were received, different techniques were 
used to select those that would be funded. In one state, staff members selected 
projects that "they feel wilMK the best" without any formal attempt at competitive 
ranking. In another, a unit outside of the regular vocational education staff ranked 
each of the proposals, using a standard rating form. The vocational education staff 
then funded the projects in the order of ranking. In one state, the SEA staff exercised 
the initiative in getting projects started. Those LEAs that wanted Part D funds 
notified the SEA of their interest. The SEA staff then presented the LEAs it selected 
with the particular projects it wanted developed. 

The degree of SEA management of local projects also varied considerably, ap- 
parently as a direct result of the interest that the SEA had in any given project. In 
states where only a few projects — all of particular interest to the state — were actual- 
ly funded, project monitoring was likely to be intense. In stntes funding a large 
number of projects, there was only token monitoring with only one or two visits a 
year. 

SEA dissemination strategy also seemed to vary with its funding strategy. 
States that funded only a few carefully developed projects were likely to be interest- 
ed in developing high-quality curriculum packages or exemplary project models, 
which could be applied in other LEAs. However, states that attempted to fund many 
projects, in order to involve as many LEAs as possible, devoted less attention to 
developing exemplary packages, and spent most of their effort on simply promoting 
the career education cause. 

Budget. The Part D program has been funded at a stable level of $16 million — 
far below its $75 million authorization level. The legislation requires that the Com- 

^ Applicants sent proposals simultaneously to USOE and to the state board of education The states 
had 60 days to reject any proposal sent to them From the remaming proposals, the federal program office 
staff and 16 outride reviewers selected the grant4?es in each state. 
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missioner of Education allocate $200,000 to each state. In addition, he is required 
to allocate the remainder of the appropriation to the states in proportion to the 
population in each state between the ages of 15 and 19. The Commissioner and each 
SEA then divide the sum allocated to each state in half and administer the halves 
independently. 

In FY 1970, the Part D program funded its first round of projects. The federally 
administered Part D program funded one project in each state for a maximum of 
three years with budgets generally between $100,000 and $150,000 per year. The 
federal contribution to the budgets remained constant in each of three years, and 
grantees were not required to provide any matching funds. However, they were 
required to state in their proposals how they expected to finance the projects after 
the expiration of the federal grant. Because the level of funding has remained stable, 
the program initiated few other new projects until the expiration of the first-round 
projects in FY 1973. 

Right-To-Read 

The Right-To-Read program, under the authority of the Cooperative Research 
Act, has developed a demonstration strategy for improved reading practices in the 
schools/ Right-To-Read designed a prototype problem-solving approach as a model 
for local districts to use in changing their reading programs at the school level. 
Right-To-Read supports a number of projects in local districts to demonstrate the 
approach. These projects, the first activities funded by Right-To- Read, were the ones 
we investigated for the change agent study. 

Guidelines. The Right-To-Read local implementation strategy, called the 
"School-Based Plan of Action," prescribes the kind of innovation that a school 
district is expected to undertake, a planning process, and organizational guidelines. 
The key elements of the Plan of Action are: 

• Project schools should implement some form of the diagnostic/ prescriptive 
approach to reading based on teaching by objectives that allows flexibility 
in the actual choice of curriculum and instructional methods. 

• Project schools should attempt a total approach to reading improvement. 
Rather than changing one or two components of the school reading pro- 
gram, the program advocates a whole series of interrelated changes, such 
as introducing new instructional methods, new curriculum materials, par- 
ent involvement, a reading center, and specialized staff. 

• hi each school all teachers and students, whether or not the students have 
severe rpading problems, should be involved. This is called the whole 
school concept. 

* Two additional components of the Right-To-Read program were not included in the change agent 
study 

1, Community-based projects. 

2. Right-To-Read works with the SEAs to coordinate existing slate and federal reading improvement 
funds and to develop the SEAs' capacity for training local educators in methods of planning and 
implementing leading improvement programs. Right-To-Read provides the states with technical 
assistance, and a small grant of funds for the administration and conduct of training programs for 
local educators 
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• In each school, the principal should be the project director and snould be 
fully responsible for project decisionmaking and management. 

• Each school should appoint a Unit Task Force consisting of a central office 
staff member, the principal, two teachers, two parents, and, optionally, the 
school librarian to plan the project and oversee implementation. 

' • Each project school should decide on its own reading improvement pro- 
gram and plan it by following an Ihstep planning process kit designed by 
Right-To-Read staff. This planning process begins with a needs assessment 
(which has also been laid out in a kit form) and includes steps to select 
project objectives, instructional materials, diagnostic instruments, instruc- 
tional components, personnel, in-service training, and the project budget. 

• Each project should emphasize staff development by spending 85 percent 
of the total budget on in-service training and other training activities. 

In addition, Right-To-Read provided each project with technical assistance from 
technical assistance teams located at five sites across the country. Members of these 
teams visited projects periodically to help with planning, in-service training, and 
problem solving. Team members were specifically trained in the 11-step planning 
process and were supposed to work closely with projects during this phase of activity. 

State Participation* Although the states are involved in other components of 
the Right-To-Read effort, management of the Right-To-Read demonstration school 
projects bypass the states entirely. 

Budget* The funding of the Right-To-Read program has remained stable at the 
relatively low level of $12 million. A portion of these funds has been spent on 
projects in both local school districts and communities to generate model reading 
programs that will be useful demonstration sites for SEAs and LEAs developing 
their own reading improvement programs. The school district projects are of two 
types, school-based projects, which are three-year grants of approximately $40,000 
per year to a single school in selected local districts; and large-city projects, which 
are three-year grants of $100,000 per year to groups of several schools in each of the 
21 largest cities in the country. 



PROGRAM EFFECTS ON PROJECT DESIGN CHOICES 

Federal change agent programs differ from each other in their management 
strategies and substantive focuses. What effect did these differences have on project 
design choices? What differences were there between Right-To-Read pi ejects and 
reading projects funded by Title III? Or between Vocational Education projects and 
Title III career education projects? Or between Title VII projects and Title III 
bilingual projects? If the answer is "none,*' then federal policies aimed at shaping 
change using the instrument of administration guidelines may be having little 
effect. 

These questions can be examined by determining which design choices were 
characteristic of projects funded by the different programs. Table 8 presents the 
results of a statistical analysis (multiple linear regression) of the relationship be- 
tween each educational method and the programs controlling for project resources, 
implementation strategy, and school characteristics. The standardized regression 



37 



Table 8 



FEDERAL PROORAM EFFECTS ON EDUCATIOKAL METliODS CONTROLLING FOR PROJECT ,CHAR.\CTERISTICS 



Project Design Choices 



Resources 
Funding 

NuDi)er of students served 
Funding per student 



Itapleroentatlon Strategy 
Percentage of planning 
Staff training time 
Value of meetings 
Teacher not participating In 

Implementation decision 
Teacher free to alter 

project design 
Materials not developed 

locally 
Project located In special 

unit 

Project covers high and 

elementary schools 
Overall teaching change 

required 
Change In specific teaching 

technique required 
Extra effort required 

(principal's view) 



Standardized Regression Coefficients for Educational Methods 



Behavioral 
Modification 



.12 
. 16 



Enrici".rnent 



School Characteristics 
Kutnber of students In 
school 

Percentage of Llack students 

In school 
Percentage of Spanish 

students In school 
Elementary school 



Federal Ptograps 
Right -To- Re ad 
Bilingual, Title VII 
Federal Vocational Education 
State Vocational Education 



Title III Focus 
Reading 
Bilingual 
Career Education 



-.01 
..01^ 



.01 
-.01 



. 16 
.01 

.15 



.00 
.00 
.01 



.05 


•07, 


06 




.00 


.08 


.05 


.06 


-.03 


.05" 


** 

-.14 


-.15 


A 

10 


-.09* 


-.03 


-.01 


.03 


-.04 


* 


-.10 


-.07 


.05 



Classroom 
Organizational 
Change 



.15 
.06 



Intensive 
TradltlT)nal 
"staffing 



.06 
-.00 

-.09 



•00,^ 

.15 

.01 

A 

.13 
.11 

-.05 
.02 
.03 
.06 
.03 

A 

.07 



05, 
.10 



.04 
.02 
.05 



.l^A 
.21 



•io, 

.14 



.00 

A 

-.14 



-•12, 
.11 



.03 
.07 

A 

.08 



-.02,, 
•l^A 

■r 

.04 
01 

A 

11 

A 

.16 
.00 
-.07 
-.06 
-.03 



Administrative 
Change 



.00 
.06^^ 



.03 
.04 



.01 
•02, 

.08 



.26, 
.11' 

A 

.11 



.02 
•03, 

.07 



•13, 
-.12 
-.04 



.16 
-.06 
.01 

.01 

-.06 

A 

-.16 

A 

-.09 
.06 
.07 
-.02 
-.05 



.15 



-.13 



.06 

.00 



. 12 
.03 

A 

.12 



.11 
.02 
.04 



R^/adju9ted R^ ^ 



.13/. 09 



.3//. 34 



.17/. 13 



.22/. 18 



.14/. 09 



NOTE: An asterisk (*) indicates significance at the .10 level; and a double asterisk (**) Indicates 
significance at the .01 level. 

^Thls coefficient is interestingly different from that of Table B-1 in App. B. 

^^All equations are significant for the F-statlstlc (24.484). Although the number of observations Is 
509, the number of projects Is 287. The classroom level Is the unit of analys^ for this table to allow 
for'classroom and school variation In project design and Implementation. 
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coefficients (beta weights) indicate how much weight each independent variable has 
on the degree of educational method (when all other variables are controlled). Ii 
particular, the coefiicients for Right-To-Read, Bilingual (Title VII), and Vocational 
jfiducation measure the amount of differences in each of the program's educational 
methods from the average Title III project. Similarly, the weights for Title Ill- 
Reading Focus, Title Ill-Bilingual Focus, and Title Ill-Career Education Focus mea- 
sure the shift up or down in levels of each educational method dimension for these 
three types of project focus, as compared with residual Title III projects with some 
other focus.* The and F-statistic measure the net extent of association between 
the independent variables and each educational method. 

By inspection of the coefficients in Table 8, let us first investigate which educa- 
tional methods were linked with which federal pro^rams.^ Projects supported by 
Right-To-Read were likely (compared wi^h the average base Title III) to involve 
intensive traditional staffing fh a major way and^.beh^yioral modification and top- 
down administrative change! Title VIT, Bilingual projects were associated with en- 
^-pkhment, as one would expect^.anSlntensive traditional staffing. Vocational Educa- 
tioh^projects were likely to employ eprichnient, and to a lesser extent behavioral 
modification and intensive traditional Staffing. Title III projects were so diverse that 
no educational method was more likely than any other; however, projects involving 
a high degree of classroom organization changes were more often associated, with 
State Title III than with other programs. 

These significant associations indicate that the various federal programs affect- 
ed project choices about educational methods in different ways. However, the nature 
of this influence hinges on the interpretation of the issue of programmatic character- 
isiics versus substantive focus: Does a federal program matter for the educational 
methocfology of an innovative project because of its administration, guidelines, and 
funding, oc-becaus^^f the proj^lem area or "focus" that programs tend to carve out 
for them^selves? WhSh we ^r the Right-To- Read, Bilingual-Title VII, and Vocation- 
al Education programs with, respectively, the Title ill projects focusing on reading, 
bilingual education, and career education, we have a natural test of the relative 
importance of federal program characteristics versus project focus. If lederal guide- 
lines matter more for tfie level;of use of the various educational methods than does 
focus, the three Title fli groups would have weights that differ from thf^ three other 
programs. If substantive focus^ apminate^ithe'pattern of relationships with me- 
thodology, the signs and weights of the.Titl^III groups would "match up'' with the 
programs of ^corresponding focus. . - ^ ' ^ 

Table 8 suggests that insofar as design decisions about the type and degree of 
educational methods were concerned, both prc^ram characteristics and substantive 
focus played important though different roles.%igl^-To-Read but not Title Ill-Read- 
ing was significantly related to the usajjf behavioral modification techniques. Thus, 
intensive use of behavioral modificat^n wasinfluenced by Right-To-Read guidelines 
rather than the substantive focus ofYfeadmg. SimUartemarks hold for Right-To- 
Read's influence on the level of intensive traditional staffing. The equivalent signifi- 

* As noted in Sec. I. since the purposive natu e of the sample does'not permit reliable estimates of 
the "absolute" association between any particula federal program and educational methods, the federal 
program effects are measured relative to the base of Title Ilf projects other than reading, career educa- 
tu)n, and bilingiaal. ' ' ^ . 

• Appendix B presents an analysis of the relationship between educational method and the indepen- 
dent variables excluding the. federal programs. 

- ■ \ 
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cant coeflicients for Right-To-Read and Title Ill-Reading on administrative change 
suggest that top-down reading curriculum changes occur for projects in both pro- 
grams.^ 

The importance of guidelines and federal program characteristic^, is also 
demonstrated in career education where, as Table 8 shows, the co^icients of the 
Vocational Education program are significantly greater than the corresponding 
Title Ill-Career Education for the use of behavioral modification and intensive tradi- 
tional staging. The evidence thus suggests that in the absence of guidelines, adminis-^. 
tration, and funding ^^'^r Right-To-Read and Vocational Education, behavioral 
modification and inten: . ditional staffing would have been less oflen employed.' 

In contrast, the substantive fociis was highly important for enrichment. The 
paired coefficients, particularly Vocational Education and Title lll-Career Educa- 
tion, reveal comparable associations and were all significantly different from%^>o. 
Similar remarks hold for classroom organization: Career education rarely involved 
jlassroom organization changes whether sponsored by Title III or Vocational Educa- 
tion.^ '\ ^ ' ^ 

In summary, the guidelines, program characteristics, funding, and ^bstantive^ 
priorities of federal programs did affect project design, but only in a partial way. 
Whether local districts were seeking opportunities for federal money or were at- 
tempting to solve ther problems by initiating innovative projects, it seems clear that 
some educational methods were more likely to be employed than others because, 
some federal programs-^either in their administrative guidelinerorin their focus- 
fostered these methods. Thus, in the absence of the-Right-To-Read program, behav- 
ioral modification techniques and concentrated traditional staffing might be less 
likely to be adopted by locally initiated reading project3. For Vocational Education, 
the existence of a federally funded program— and the funding opportunities afforded 
by it— induced higli levels of enrichment activities to be pursued only as long as the 
federal money existed. However,^he federal change agent programs did not greatly 
influence the design of such locally conceived pr^ojects as innovations in classroom 
organization. 

' The same conclusion held for the importance ofTitie VII on mtensive traditional stiifling In the 
absence ofTitle Vil, bihngual projects would have been less likely to employ intensive traditional staffing 
" Table B-1 of App. B presents the results of an analysis that is identical with the analysis of Table 
8 except that the appendix table does not include the federal programs as independent variables '^o 
general observations about program im^ lications are suggested by con^paring Table » with lable B-1 
Aside from some exceptions to be discussed shortly, the sign, significance, a- strength of association 
between implementation strategies and educational methods remain about the same (vhen the ( shift ) 
effect of federal programs and focus are taken into account. This indicates-although not con elusive ly— 
that the patterns of implementation strategies and choices are linked to educational methods independ- 
ently of the source of federal funding. For example, classroom organization projects have common 
strategic elements regardless of the program that funds them. 

Another obvious but sometimes ignored effect of federal policy is suggested by the several changoe 
n coefficients when federal programs are introduced into the analysis. Ck)mparing Table 8 with Table 
B-1 the percentage of black students at a project school loses its strong iissociation with intensive 
traditional stafTini nd behavioral modification when federal programs are controlled for This statistical 
effect which aris.. from the relationship between Right-To-Read and percentage of black students, 
reflects a targeting policy used by the Right-To-Read program of giving grants to large cities; some of these 
grants were given in c^ses where the districts had not developed a project design and k^ame viewed as 
grams m aid comparable wiih Title I monev. The loss of significance for percentage of Spanish students 
m a school on intensiv&^raditional staffing is due to the obvious correlation between Bilingual programs 
and their mam target tn-oup (in conj'inction with the Title VII emphasis on intensive traditional staffing) 
Tb"s. the evidence indicates that target populations who* traditionally have had lower priority in the 
allocation of local funds have reteived some special benefits. However, in the abser^c of effective federal 
monitoring, there is no guarantee that these categorical funds will be used for their intended purpose 
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Thus, federal policy using the instrumentaof guidelines and categorical priori 
tieh tied to diflerential funding may have some limited leverage on the design choices 
made by local schoo! districts. In particular, the federal programs can influence how 
intensively sonie educational treatments are tried. However, federal policy instru- 
ments of the type employed by the change agent programs may have little influence 
on the frequency with which some types of educational methods, particularly class- 
room organization change, are initiated by the local districts in order to meet their 
own needs and priorities. 



IMPLEMENTATION DIFFICULTIES 

Many types of problems arise in implementing innovations. Some difficulties 
involve the substance of the innovation itself For example, an innovation employing 
a new teaching technique may lead to problems because teachers find it difficult to 
learn the technique. Other problems can be attributed to inadequate planning, for 
example, the lack of space, equipment, or materials essentjal to the project. Some 
innovation arouse faculty resistance or community opposition. And some projects 
suffer from poor leadership. To deveLp a comprehensive picture of these implemen- 
tation difficulties, we asked teachers \vhat problems they had to deal with during the 
life of the projv^ct. The catalogue of problems along with the frequency o.^^'esponses 
are shown in Table 9. '^^^^ 

Behind these responses lie revealing stories about the process of change that 
only fieldwork of the type reported by Vol. Ill can document. Our purpose here is 
to describe the patterns of difficulties that arise during implementation for different 



Table 9 

IMPLEMrNTATION PROBLFIMS AS CITED BY TEACHERS 









' ' ' 




Number of Times 




Mentioned 


(ioals not sufficienLly (iefined 


170 


Techniques compl icaled or unclear 


161 


Feactiers mifamiliar with macerlais and mectiods 


33A 


\ adequate space, materials, or equipment 


328 


Unanticipat"d requirements 


116 


Unrealistic goals or schedule 


96 


Teacliers already overloaded 


187 


(Leadersiup or management conflicts) 


85 


Faculty or staff resistance 


107 


Parental or community opposi t ion 


55 


Delay or reduction in funding 


113 


Ot}ier problems 


8 


No ^>rLbiGms 


A8 



NOTE: Number of responses = 68^*. 
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types of federal programs. To this end, we analyzed statistically how much each 
problem cited by teachers contributed to their own evaluation of the difiiculty of 
project implementation. Specifically, we regressed ths difficulties of implementation 
(as defined in Sec. H) on the problems cited by teachers. Table 10 presents the 
results— a comparison of the eflects of the problems for the various federal pro- 
grams.^ 

As Table 10 indicates, the problems may be grouped into four types. First, there 
were problems associated with the substance of the project's goals or educational 
methods, namely, goals not sufficiently defined (i.p., lack of specificity of goals), 
tech niques complicated or unclear (which is a dimension of complexity), and teachers 
unfamiliar with materials or methods (which is dependent on the relative newness 
of the innovation). Second, some problems were a consequence of deficiencies of the 
prior plans for implementation, namely, the inadequate space, equipment, or 
materials, unanticipated requirements, unrealistic goals or schedule, and teacherb 
already overloaded with other commitments. Third, some projects provoked difficul- 
ties as they became impleniented in the given institutional setting, namely, leader 
ship or management problems, faculty or staff resistance to the project, and parental 
or community opposition. Fourth, delay or reduction in funding caused problems on 
•some projects. In addition to this classification, other idiosyncratic problems, as well 
as the absence of any mention of problems, were coded. 

Before companng federal programs, we will discuss the results for all programs 
combined, as shown in col. 1 of Table 10. The most striking finding is the significance 
of deficiencies in prior planning- The four types of prior planning problems in- 
dividually and jointly had very significant effects on the teachers' perception of 
difficulty. 

To -show how large these effects were, as well as to illustrate the interpretation 
of the regression coefficients, let us consider the coefficient of .31 for 'Teachers 
already overloaded" indicated in Table 10. Since the independent variables'are 
dichotomous, the coefficient means that a project for which teachers say that they 
were already overloaded adds a .31 increment of difficulty as compared with a project 
not having this problem; in other words, the average difficulty of all projects if 
teachers did not mention any problems was 1.98, (on a scale of 1 very easy, 2 fairly 
easy, 3 somewhat difficult, and 4 very difficult), and projects experiencing teacher 
overload— but no other problems—would have an average difficulty of 2.29. Another 
ii.terpretation of the regression coefficients is as a set of linear weights; in these 
terms, the perception of being overloaded contributes .31 to the teachers' overall 
feeling of the difficulty of implementing the innovations. All four of the types of 
problems arising from deficiencies in prior planning of projects contribute 1.07 to 
difficulty (.27 \ .25 4 .24 I .31), which is the difference between an easy project 
and a difficult one.*^ 

• Because of the nonrepresent^tive nature of the sample, it is not useful w analyze the frequency of 
responses per se shown in Table 9 However, it is appropriate to examine the conditional question' Oiven 
that certain problems arose m a project (according to the teachers), what effects did these problems I e 
on the teachers' overall feeling about the difficulty of implemontation'> The regression results shown by 
Table 10 provide an answer to this conditional question The independent variables are dichotomously 
coded U e . they are dummy variables) 

A T-test indicates that the sum of these problems is not significantly different fron. 1 0 The various 
problems are, of course, intercorrelated However, neither the pa.rwise correlations of problems nor 
multiple correlations of any problem with all other problems are so high as to preclude reasonably 
accurate estimates of the contribution made by each problem (The regressions for F?ight-To-Read and 
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That deficiencies in prior planning foi an innovation contributed to the difficul- 
ty of implementation is not unexpected. However, lest this finding be misinterpret- 
ed, a distinction about different types of planning should be noted. The nature of 
prior planning can be conceptualized into three ideal strategies that are based on 
usually implicit notions of how change occurs. First, there is a technological notion 
that implementation difficulties are not anticipated or planned for, and an innova- 
tion is assumed to be more or less automatically put in place. Planning based on this 
technological view is usually limited to logistical considerations. The implementa- 
tion of such technolc.^ies as computer-assisted instiuction oflen has followed this 
model without serious consequences. But some local designers of more complex 
innovations have ignored the need for prior planning about implementation. 

A second approach recognizes that innovations cause difficulties but assumes 
that rational planning can anticipate not only the problems, but also their solutions. 
The third strategy takes an adaptii^e,viev/ in which unanticipated requirements are 
expected, but rather than trying to layout all possible contingencies, prior planning 
takes the form of trying to build problem-solving mechanisms into the project de- 
sign. The planning fbr any particular project may be characterized in terms of the 
mixture of these elements in its design. 

Our purpose in drawing these distinctions here (Vol. Ill will illustrate them in 
detail) is to warn against a too simple inference from the data analysis: Since 
deficiencies in prior planning strongly influence the difficulty of implementation (as 
perceived by teachers), more planning would solve the problem. Undoubtedly, many 
projects suffer from a technological assumption and consequently do not plan suflSx 
ciently for new materials needed in day-to-day implementation. But the cure is not 
necessarily the rational planning model, which can lead to unanticipated require- 
ments and unrealistic goals and schedules. Instead, a flexible plan that allows for 
adaptation and the local development of materials may produce fewer true difficul- 
ties in implementation. 

Of the group of problems associated with the project's goals or educational 
methods. Table 10 shows that both the lack of specificity of goals and the complexity 
of techniques contributed significantly to the teachers* perception of diflSculty of 
implementation. However, the newness of the materials was not significant, al- 
though teachers cited problems caused by their unfamiliarity with methods and 
materials more than any other probleni. Apparently, teachers did make the distinc- 
tion between newness and fuzziness or complexity. 

Of the group of problems relating to the institutional setting. Table 10 sug- 
gested that parental or community opposition was very important, whereas the 
other problems were not significant.*^ This finding will be discussed in more detail 



for Federal Title III contain several poor estimates due to multicollinearity as indicated by inflated 
standard errors ) Rather than display the six pairwise correlation matrices, an indication of the low level 
of intercorrelations are the following figures for the highest correlation among the problems for each 
federal program all programs. 26, State Tale III, 25. Federal Title III, 39. Bilingual. 26, Federal and 
State Vocational Fducation, .33; and Right'To-Re/id. 44 

We expect teachers to be birsed in their report of faculty resistance to innovatiortr Consequently, 
the estimate of the effect of faculty resistance is likely to be lower ttian the estimate for other participants. 
Usmg the <;nswers of project directors on the same questions and running regressions sir»u!ar to those 
of Table 10, we Rnd that project directors are somewhat more likely to report faculty resistance — 
although not extraordinarily more— and that such resistance has significant weights for Bilingual and 
Fe<ieral Title III programs, but not for the other piograms Some of the typical faculty problems on 
Bilingual are documented in Vol III Some Federal Title III projects involved administrative or school- 
wide changes \n schedules and integrated curricula that can provoke faculty resistance 
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when we compare the various federal programs.*^ The final category of delay or 
reduction in funding also affected the difficulty of implementation, as one would 
expect. 

Thus far, the analysis has treated problems arising in projects independent of 
their source of funding. We now turn to comparisons of the effects of problems on 
the difficulty of implementation for the various federal change agent programs. 

State Title III 

Because federal change agent programs have distinct guidelines, requirements, 
and substantive priorities, we expect that the local innovative projects might well 
have implementation problems that reflect their funding source. Thus, the locally 
initiated, highly ^competitive State Title III projects might be expected to avoid 
severe problems resulting from unclear or unfamiliar techniques, simply because 
their techniques were locally chosen. The data shown in col. 2 of Table 10 support 
this expectation: State Title III teachers who reported complicated, unclear, or 
unfamiliar techniques were not significantly more likely to encounter implementa- 
tion difficulties as a result of this problem. 

By contrast, the problem of unclear goals was a significant source of difficulty 
for teachers in State Title III projects. The initial goals of these projects tended to 
be ambitious or ambiguous. The ambiguity may have stemmud from the competitive- 
ness of Title III awards, which may have elicited vague grantsmanship claims in 
proposal writing. The overly ambitious goals may have reflected the opportunity 
provided by Title III for districts to take a chance on highly innovative and complex 
projects. But a lack of goal specifidty can later create confusion and conflict. The 
freedom of LEAs under nonc/itegorical programs such as Title III may thus cut two 
ways. It may reduce pote'ntial implementation problems arising from the selection 
of unclear techniques, but it may increase future stress if goals are ambiguous and 
overly ambitious. 

TabJc 10 shows that prior plan ning deficiencies created significant problems for 
State Title III teachers. Teacher overload, with a coefficient of .32, impeded im- 
plementation, even in locally conceived projects. An advantage of local project initia- 
tion may be the potential for gaining broad institutional support of projects that 
focus on central problems of the LEA. Yet if teachers feel too burdened by existing 
reponsibilities, they may be unable or unwilling to support any innovation. Other 
planning problems, especially inadequate facilities (.20) and unanticipated require- 
ments!. 26), were important sources of difficulty. An ability to correct these problems 
promptly once they are discovered could be of great value to the implementers of 
a new project. 

Although faculty resistance had a small (.12) and unstable relation to im- 
plementation difficulties, Table 10 indicates that parental opposition increased (by 
.30) the severity of the difficulties. It is notabla-that problems of communication- 
such as parental opposition, lack of goal clarity, or unanticipated requirements— 
pr- ed more significant than technique or funding problems (both with insignificant 
.13 coefficients). In a diverse program such as Title III, the issues of group dynamics 
and cohesion seemed persistently relevant to the implementation of change. 

Leadership or management problems wore not listed in original survey question, but were 
coded from the "Other problems" category This omission undoubtedly caused an underestimate? of the 
problems of leadership or management, '^0 
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Federal Title III 

Federal Title III projects difTered from State Title III projects in thdr mean 
funding levels and target group sizes, although both were designed and organized 
by LEAs. Proposals for Federal T^le HI support typically requested large-dollar 
grants to be spent on programs aim^ at a large number of students. In addition, 
Federal Title III was the most likely of all change agent programs to focus on school 
organization Changes.*^ This systematic difference between the programs may ac- 
count for the different kinds of implementation problems they faced. 

Column 3 of Table 10 shows that the problem of complicated or unclear tech- 
niques was significantly related (.43) to difficulties for Federal Title III funded pro- 
jects. Despite the local initiation of the project, it appears that the broad scale of 
operations for relatively large and expensive projects m^y have interfered with the 
teachers' clear understanding of new techniques. Other substantive issues— goal 
definition and unfamiliarity— were small and insignificant as problems. Projects 
that attempted schoolwide organization change may have had simple goals (e.g., an 
"integrated curriculum.") but very unclear techniques. 

Teacher overload was the onl> large (.65) source of difficulty in the prior plan- 
ning of Federal Title III projects. The scope, in students and expenditures, of these 
innovations may have made it impossible to mesh the project with the workload of 
all project teachers teachers w^o reported this problem apparently found it to be 
a major obstacle to implementing the new program. 

Since the thorniest problems in Federal Title III seem to be related to the 
substantial scale of some of the projects, it is easy to assume that reducing average 
project size would ,be the proper antidote. The data do not support this solution. 
Federal Title III projects had considerably fewer problems that were significantly 
related to difficulty with implementation than other programs. The institutional 
support, momentum, and overhead provided by large grants may be an administra- 
tive and planning benefit that tends to outweigh the difficulties of large project size. 
Because a substantial amount of the variance in implementation difficulty for Sec- 
tion 306 projects is jointly explained by problems arising from understanding tech- 
niques and from teacher overload, considerable advantages may accrue from at- 
tempts to increase the interaction between project managers and the many teachers 
participating in the project. 

Bilingual, Title VII 

Bilingual projects were among the most difficult to implement, as the high 
constant term (2.26) in col. 4 of Table 10 suggests. Inadequate materials (.52), teacher 
overload (.38), and unrealistic goals or schedule (.32) all contributed significantly to 
project difficulty. These problems, particularly those of obtaining bilingual cur- 
riculum materials and qualified staff, may have stemmed from the relative newness 
of interest in bilingual education. But they also may reflect the complex and highly 
ambitious nature of changes attempted by some bilingual projects. 

Vocational Education 

Column 5 of Table 10 shows that only two problems encountered by teachers 

61 

»^ See Sec. II and App B 
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were significantly related to the difficulty of implementing Vocational Education 
projects. These problems, complicated or unclear techniques and parental or com- 
munity opposition, account for most of the variance explained in implementation 
difficulty (R^ .42). Their large effect requires an explanation based on the setting, 
rather than on the substance, of career education projects. 

The analysis of fieldwork conducted for Vocational Educational projects (see 
Vol. Ill) suggests that these projects rarely enjoyed support from the local district. 
Because little serious proposal development was required by the enabling legisla- 
tion, vocational projects were often treated as grant^in aid. Moreover, many projects 
were initiated as a result of minority group demands (in the Model Cities cases) or 
thedesiies of a small group of professionals to increase vocational activities. Typical- 
ly, neither ^^ituation was conducive to gaining district-level support for the resulting 
project 

In this unstable situation, innovations may have been vulnerable to dissension 
and poorly targeted techniques. Projects may have deteriorated in part because 
participating teachers lacked incentives to try to make them succeed. This failure 
muy not be inherently related to career education, but to the situation in which such 
projects were initiated. 

There is an odd corollary to this vulnerability. When technique and community 
problems were>not encountered, project teachers did not report implementation 
difficulties simply because little change was demanded of them. We shall reuirn to 
these hypotheses in Sec. IV. 

Right-To-Read 

Right-To-Read demonstration projects emphasize prescribed management 
strategies and diagnostic/prescriptive approaches to reading achievement. The way 
LEAs dealt with these packages may have determined how difficult implementing 
the project was in practice. First, Table 10 shows that the substantive problem of 
teacher unfamiliarity with materials was a serious source of difficulty for Right-To- 
Read projects, and for no other programs. If teachers objected to, failed to under- 
stand, or could not use the packaged reading curricula and instruments provided for 
them, the project could become extremely frustrating. This finding suggests that 
staff training may have been inappropriate in nature or deficient in amount. This 
is relevant to a thesis advanced in Vol. Ill — local material development may be an 
important ingredient in the adaptation process that accompanies serious change 
eflbrts. 

Second, the general problem of prior planning was quite strongly related to 
implementation difficulties. Unanticipated requirements (.39), inadequate materials 
(.34), and teacher overload (.26) were all sources of difficulty. Ta' .*n together, these 
factors suggest that the rational planning model implicit in these projects may 
retard the flexibility necessary to deal with day-to-day problems. 

Finally, when parental or community opposition was present, it could substan- 
tially impede implementation (as indicated by the coefficient of .66). The importance 
of reading instruction in the educational mission of schools makes reading a very 
salient policy area for parents. If they disapproved of or misunderstood the Right-To- 
Read approach — perhaps interpreting it as remedial in intent— teachers may have 
encountered stiff and continuing resistance to the project. 
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Summary 

The problems that most seriously affected implementation stemmed from two 
sources. First, innovation is intrinsically a disruptive process. Problems attributable 
to inexperience of project participants in planning for change and adapting to its 
demands were bound to arise. Second, projects encountered difficulties that reflected 
the selection mechanisms, administrative guidelines, and substantive priorities of 
the federal programs. In particular: 

1. State Title III projects experienced difficulties because of insufficiently 
defined goals and inadequate prior planning. 

2. Federal Title III projects had topnlown problems of complicated or unclear 
techniques and problems of requiring more additional work than teachers 
could handle. 

3. Title VII Bilingual projects lacked bilingual materials and staff. 

4. Vocational Education projects experienced difficulties gaining support 
within the district and within the community. 

5. Right-To-Read projects had difficulties of implementation that arose from 
the teachers' unfamiliarity with prescribed materials and methods, and 
from adaptation requirements that were not anticipated by the program 
management strategy. When parental opposition existed, Right-To-Read 
projects experienced considerable difficulty. 

Thus, the patterns of difficulty associated with the implementation of different 
federal programs demonstrated that the management strategies have subtle influ- 
ences beyond the initiation stage. Insofar as federal change agent programs affect 
initial project design, they also may create specific barriers that local innovations 
must overcome. 



IV. IMPLEMENTATION OUTCOMES: CLASSROOM 

LEVEL ' 



We have argued so far that differences between federal programs were partially 
played out in project design choices and problems encountered during project im- 
plementation. To what extent did these differences also affect implementation out- 
comes? And if these differences in programs had only marginal effects, what other 
factors account for the variation in implementation outcomes? 

To answer these questions, many evaluations of educational innovations have 
implicitly assumed a naive input/output view of change. The treatment — namely, 
either the educational technology or the federal change agent program— is consid- 
ered a potentially significant input to the "black box" of the school system from 
which come output^ usually measured as student outcomes. This approach has not 
proved very useful for two reason^. First, variations in student outcomes have not 
been consistently related to variations in treatments, once nonschool factors are 
held constant. Second, because of its atheoretical character, the evaluation litera- 
ture does not permit us to generalize from particular project assessments or even 
to specify the elements of the black box that have been ignored. 

This section begins to unpack the black box. We will analyze the effects on 
implementation outcomes (as previously defined) of both the treatment and the 
elements that are usually ignored — implementation strategies and institutional 
setting. 



GROSS COMPARISON OF PROGRAM AND TREATMENT ON 
IMPLEMENTATION OUTCOMES 

To what extent did the differences between the federal change agent programs 
result in differences in the implementation outcomes of innovative projects?* Table 
11 compares the mean of the implementation-outcomes scores of projects funded by 
the various federal change agent programs.^ Aside from the difficulty of implemen- 
tation, the federal programs did not differ substantially from one another on per- 
centage of goals achieved, change in teacher behavior, or extent of implementation 
as laid out.^ 

This finding suggests that federal change agent programs had approximately 
equal effects on implementation, despite their different management strategies. A 
simple and yet appropriate (as we shall later argue) explanation for equivalent 
program effects is that the policy common to these programs had limited influence 

' Federal programs implicitly or explicitly serve many objectives in addition to the ones considered 
in this quantitative analysis or m the larger Hand study Our comparisons are thus iimitt^i and should 
be treated cautiously by the reader. 

^ The absolute value of these numbers is probably biased, as previously explained However, if we 
assume for the sake of this section that projects on all the programs are biased in the SJime way (a 
reiisonably plausible assumption) the comparisons made above are appropriate. 

^ Section III compared the change agent programs in terms of problems arising during implementa- 
tion and the consequent difficulty of implementfUion. 

CI 
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Table 11 

PROGRAM COMPARISON ON IMPLEMENTATION OUTCC^ES 



Average Score (standard errors) 



Federal Program 


Percent- 
age of 
Goals 

Achieved 


Teaclier 
Change 


Difficulty 
of 

Implementat Ion 


Implementation 
as 

Laid Out 


Title III (State and Federal) 


69.9 
(1.3) 


3.02 
(.05) 


2.47 
(.04) 


3.50 
(.03) 


Right-To-Read 


70.1 
(3.0) 


3.10 
(.12) 


2.47 
(.10) 


3.60 
(.08) 


T^^^^> VTT Rilincual 


71.3 
(2.9) 


3.02 
(.11) 


2. 74 
(.10) 


3.57 
(.07) 


Vocational Education (State 


75.4 
(3.3) 


2.82 
(.13) 


2.10 
(.11) 


3.51 
(.08) 




.00 


.00 


.04 


.00 


Significance of F-statistic 
(3503) 


.49 


.44 


.00 


.65 


Significance of T-test that 
Title III ^ Right-To-Read 


.95 


.54 


.98 


.27 


Title III ^ Bilingual 


.66 


.98 


.01 


.43 


Title III ^ Vocational 


.12 


.16 


*** 

.00 


.95 


Right-To-Read ^ Bilingual 


.78 


.62 


.05 


.80 


Right-To-Read i Vocational 


.24 


.12 


.01 


.45 


Bilingual ^ Vocational 


.35 


.26 


.00 


.60 


Range of dependent variable 


0-100 


1-4 


1-4 


1-4 



NOTE: An asterisk (*) indicates significance at the .15 level; a double 
asterisk (**) indicates significance at the .10 level; and a triple asterisk 
(***) indicates significance at the .01 level. Coefficients of programs are, 
of course, all significantly different from zero but not from the overall 
average of all projects. 
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on implementation— factors other than federal policy affected implementation in 
major ways. 

Because the change agent policy common to the programs had limited influ- 
ence, each separate program could affect project implementation only marginally 
Within this latitude of marginal effects, some significant differences between pro- 
grams could be discerned, 

The significant differences between the programs revealed by Table 11 involved 
Vocational Education. Projects funded by Vocational Education programs had the 
highest average reported percentage of goals achieved and Title III the lowest, 
although the difference between the two is barely significant. Yet Vocational Educa- 
tion projects were, on the average, least likely to result in teacher change; moreover, 
they were reportedly the easiest to implement, particularly when compared with the 
program with the most difficult projects. Bilingual. These gross comparisons suggest 
that Vocational Education projects achieved the highest percentage of their goals 
because they were not attempting major innovations. Most such projects simply 
added career enrichment materials to the curriculum.** 

Besides the apparent differences between Vocational Education and the other 
programs, the data of Table 11 indicate that differences between the other federal 
change, agent programs did not by themselves explain the variations in project 
outcomes. This is not surprising, for these comparisons did not take into account the 
differences among project characteristics within the same program. Accordingly, it 
makes sense to compare the effects on implementation outcomes of the various 
educational methods within the federal programs. 

Table 12 presents a multiple regression analysis of the effects on the four out- 
comes of variations in educational methods (or treatments) for projects within the 
various change agent programs. The first row shows the average outcomes for Title 
III projects (holding within-program variations in educational methods at a constant 
value of zero), followed by five rows that display the regression coefficients or effects 
of varying levels of educational treatment for Title III projects. Ignoring the absolute 
estimates of the average outcomes for each program, the regression coefficients of 
each method within each progr?ni represent the amount of change in implementa- 
tion outcomes that an increase in one unit (a standard deviation) of the level of an 
educational method would induce.^ For example, the significant 1.7 coefficient of 
Title III classroom organization methods indicates that for each unit increase of 

* See App. B for quantit«'itive evidence on this point. 

' The "coefficients" of the federal programs represent the mean irnplementationK)utcome score for 
the level of educational methods at their mean value (which is zer^ standardization). For example, 

69.7 percent is the average perceived success score for Title HI prujects when the level of behavioral 
modification, enrichment, etc. are average. For such programs as Right To-Read, whose projects tend to 
have high levels of beha%ioral modification, intensive staffing, and tosome extent administrative changes, 
average success is probably better computed at higher levels of the use of these methods The following 
table computes the average success scores based on different levels of method employment to show the 
sensitivity of these average scores* 



Level of Method Used on All Methods 



Program 



Average 



I Standard Deviation 
Above Average 



2 Standard Deviations 
Above Average 



Title III . 
Right-To^Read 
Bilingual . . 
Vocational . . 



69.7 
85.6 
64.0 
75.1 



70.^^ 
78 3 
85.3 
8.1 0 



71 3 
71.0 
106.6 
92.0 




Table 12 

PROGRAM-METHOD COMPARISON ON IMPLEMENTATION OUTCOMES 



Educational Methods 
by Program 



Title III 


69. 


7 




(1. 


3) 


Behavioral toodlf Ication 


1 


0 




(I 


2) 


Enrlchnent 


-1 


7* 




(1 


2) 


Classroom organization 


1 


* 

7 




(1 


2) 


Intensive traditional staffing 




0 




(1 


.3) 


Adnlnlstrative changes 




.6 




(1 


.A) 



Right-To-Read 

Behavioral modification 
Enrlchcacnt 

Classrooci organization 
Intensive traditional staffing 
Administrative changes 



Regression Coefficients and Standard Errors 



Percent- 
age of 
Goals 

Achieved 



85.6 
(8.9) 

(3.5) 

15.0 
(5.6) 

-5.6 
(5.1) 

-3.5 
(7.1) 
* 

-8.8 
(^.1) 



Teacher 
Chani;e 



3.02 
(.05) 

-.03 
(.05) 

-.02 
(.05) 

AAA 

.12 

(.05) 

A 

.08 
(.05) 

(.05) 



3.40 
(.34) 

-.00 
(.13) 

.06 
(.22) 

-.17 
(.20) 

-.22 

(.28) 

AA 

-. 32 
(. lt>) 



Implementation 
as 

Uld Out 



3.51 
(.03) 

.02 
(.03) 

A 

-.05 
(.03) 

-.00 
(.03) 

.04 
(.03) 

.02 
(.04) 



3.33 
(.23) 

-.02 
(.09) 

-.08 
(.15) 

.15 
(.13) 

AA 

. 34 
(.18) 

-,05 
(.11) 



Difficulty 
of 

ImpleDentation 



2.45 
(.04) 

AAA 

-.11 

(.04) 

.04 
(.04) 

AA 

.07 
(.04) 

AAA 

.12 
(.04) 

AAA 

.13 
(.04) 



2.86 
(.29) 

A 

-.18 
(.11) 

.17 
(.19) 

. 14 
(.17) 

-.30 
(.24) 

AA 

-.23 
(.14) 



Bilingual, Title VII 

Behavioral codification 
Enr IcJuaent 

Classroota organization 
Intensive traditional staffing 
Administrative change 



Vocat ional Educat Ion 

Behavioral aodlflcatlon 
Enrichment 

Classroom organization 
Intensive traditional staffing 
Administrative changes 



Significance ot F-test statistic 
(23.483) 

Slgnlfl nt (.10) T-te*Jt6 on 
differences of program 
constants 



64.0 
(4.7) 

1.9 

(3.4) 

6.9 
(6.7) 

.8 

(4.9) 

A 

8.2 
(3.4) 

3.5 

it •>\ 



75.1 
(7.2) 

.6 
(3.6) 

-1.1 

(5.5) 

1.9 

(4.0) 

AA 

10.4 
(4.5) 

-2.9 
(5.6) 



2.79 
(.18) 

.18 
(.13) 

.34 

(.26) 

.04 
(.19) 

.08 
(.21) 

.13 

(,12) 



.06 



.11 



RTR>T3 
RTR^BIL 



2.50 
(.28) 

. 16 
(.14) 

.29 
(.21) 

A 

.24 
(.15) 

A 

.26 
(.17) 

-« 10 
(.22) 



3.54 
(.12) 

-.10 
(.09) 

.07 
(.17) 

-.08 
(.13) 

.09 
(.14) 

-.10 

(.08) 



3.25 
(.19) 

.06 
(.09) 

.18 
( 14) 

.11 
(.10) 

AA 

.24 
(.12) 

Atf 

-.25 
(.14) 



.06 



.12 



T3>V0C 
RTR>V0C 
RTR>8II, 



.65 



2.78 
(.15) 

.08 
(.11) 

- 15 
(.22) 

.13 
(.16) 

-.00 
(.IB) 

-.08 
(.10) 



2.00 
(.24) 

.06 
(.12) 

.15 
(.18) 

AAA 

.40 ^ 

(.13) 

-.02 
(.15) 

-.24 
(.19) 



.13 



.00 



BIL>T3 
T3>V0C 
RTR>VOC 

Bn>>voc 



NOTE: An aaterlsk (*) Indicates significance at the .13 level;- a double asterisk 

(**) Indicates significance at the .10 level; and a triple asterli^k (***) Indicates 
significance at the -01 level. 

^he general foria of the estimating equation Is ?i + ^ bjj PiE^. where Pi Is 

the l'^ program. L, is the j'^ educational rr^thod. and H «n<J ^Ij ^'"^ ^f^l ^""l m . 
Coefficients of programs (which are dummy variables) are. of course, all significantly 
dHferent from zero but a .oore relevant ^^rnyl^on Is with the average of all projects. 

* 
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classroom organization activities, Title III projects reported a 1.7 percent higher 
percentage of goals achieved. 

In terms of variations of treatments within the federal programs, Table 12 
reveals an important marginal effect. For Title III, classroom organization methods 
were positively related to teacher change and difficulty of implementation as well 
as to percentage of goals achieved. However, Title III enrichment methods were 
negatively related to perceived success and negatively related to implementation as 
laid out (probably because of the diffuseness of Title III enrichment projects). Inten- 
sive traditional staffing on Title III projects not only was significantly related to 
increased difficulty of implementation but also positively related to the amount of 
teacher change (but whatever the nature of this change may have been, intensive 
traditional staffing did not increase the percentage of goals achieved on Title III 
projects^® Schoolwide administrative changes increased difficulty of implementa- 
tion but did not affect other outcomes. 

These findings can be interpreted by recalling the nature of the Title III pro- 
gram. Since Title III innovations were locally initiated and competitively chosen, 
they included a wide mixture of projects, ranging from straightforward enrichment 
activities to complex attempts to alter classroom organization. The Title III projects 
involving sustained efforts in classroom innovations were difficult to implement but 
more likely to be successful in terms of both teacher change and percentage of goals 
achieved. Yet numerous Title III projects that engaged, for example, in simple 
enrichment or in staff development unrelated to classroom activities did not appear 
to have achieved their goals. The diversity in the scope of Title III projects thus 
seems matched by uneven outcomes. In short, the Title III management strategy 
may have yielded high risks but high returns.' 

Besides these significant differences in treatments within Title III, the overall 
results suggest that neither differences between programs nor variations of educa- 
tional methods within programs explain much of the variation in project outcomes. 
(Note in Table 12 the low level of and the level of significance for the overall F-test 
for each outcome, excepting difficulty of implementation.) Thus, these gross input/ 
output comparisons or,program and treatments need to be supplemented by other 
factors that influence the outcomes of innovations. 



FACTORS AtTECTING IMPLEMENTATION OUTCOMES 

The most important broad hypotheses in this report are that the institutional 
setting and project . ^aracteristics relevant to implementation have major effects on 
the implementation o. comes of innovative projects. Table 13 presents evidence 
that supports these hypotheses. 

In particular. Table 13 reports the results? of a multiple regression analysis in 
which the teachers' perceived success rating of a project (the percentage of goals 
achieved) is considered to depend on project characteristics (i.e., project resources, 
educational methods, and implementation strategy), on elements of the institutional 
setting (i.e., organizational climate, school characteristics, and teacher character- 

• The findings m Vol Ill's analysis of staff development projects funded by Title III aro ronsistent with 
and provide an interpret^ition for these results, Namely, although teachers were changed by staff develop- 
ment, these projects were not always tied to ongomg classroom activities, 

6S 



Table 11 

KACIORS AVrh-CriNG I'rJ<Ci:iVK!) suc^ccss 



Standardized Regression Coefficients 
for Percentage of Goals Achieved 



Independent Variables 



Pro ject Characterlst ics 

i Funding 



Resourc'jj 



Kdu^^at lonal 
me thods 



ERIC 



ImpK .nentat ion 
St racegy 



-J Number of students served 
(Funding per student 

(Behavioral modification 
Enrichment 

(Classroom organizational change 
Intensive traditional staffing 
Administrac Ive change 
/Percentage of planning 
Staff training tine 
Value of meetings 
Teacher not participating in 

implementation decision 
Teacher free to alter project 
design 

Materials net developed locaJ ly 
Project located in special unit 
Project covers high and elementary 
schools 

Overall teaching cliange re qui red 
Change in specific teaching 

technique required 
Kxtra ^effort requlreti (principal's 
view) 



All 
Projects 



\ 



Organizat icnal 
climate 



Institutional Setting 

(Kxtra effo»-t (teacher's view) 
Morale of teachers at school 
.Support from superintendent 
/Support from principal 
\ Support from faculty 
Number of students in sch'^ol 
Percentage of black students 
School charac- / school 

t ^istics ^ Percentage of Spanish students in 



Teacher charac- 
teristics 



Percentage 

school 
Hleme ntary 



school 



Teaclier e/<perience 



Federal Programs 



Right -To- Read 
Bilingual. Title VII 
Federal Vocational Education 
State Vocat ional Education 



.06" 
.06 

.08 
.08 
.06 
.03 
.06 
.06 
.00, 

.11 

A 

-.10 

.03 
.01 
-.00 

-.12 
-.06 



(2) 

All Projects i* 
Klementarv Scho^ 



/ad justed R" 
Number oi observat ion<^ 



.08 
.01 
.06 
.00 



309 



.06 

.00 

.Ob 

.05' 

.06, 

.09 

.02 

.00 

.06, 
.12 

.13 

.01 
.01 

.13 
.02 



07 


-.03 


00 


-.02 


A* 


.05, 




■2^ 


.26' 




.02 


.10, 


.11 


.06 


.05 


.06 


.00 


*** 




. 12 


.07 




-.02 


.11 




.03 


.06 



-.07 
.01 
.00 

-.01 



.19/. 11 



(3) 



Title III 



-.0^> 
.03 
.OA 

■07, 
-.09, 

-09, 
-.13 
.06 
-.02 
.OA, 
.12 

-.15 



.03, 

.09 

.0/ 



.lA 
.11 

.07 

.03 



.09, 

■21, 

■15, 

.13 

.OA 

.01 



.13 



.08, 
. lA 



.10 



.28/. 1<) 

289 



'lOJ- , Ar asterisk (*) mdirates signiTicancp ,it the .15 level; a double astensk (**) in- 
dicates ^Unifirance at the .10 level; and a triple asterisk (***) indicates significance ?l 
the .01 level. 

''^Funding would positively signlXi<.ant if the federal programs were not in these regres- 
sions, The effects of other varial.les remain essentially unchanged when federal pror.rams and 
educational methods are removed from the analysis. 

CO 
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istics), and on the federal program supporting th' project. The table displays stand- 
ardized regression coefficients (beta weights) fo; three situations, all projects in the 
sample, projects in elementary schools, and projects funded by Title III. Each coeffi- 
cient represents the effect of its factor on percentage of goals achieved, statistically 
taking into account the other factors in the analysis. 

Before detailing the findings suggested by Table 13, ? general observation is in 
order. As Sec. I warned, this analysis must be treated cautiously, taking due note 
of weaknesses in both measurement and in our ability to specify the variables and 
their relationships in *he best possible way. Yet even being cautious and discounting 
the exactness of the estimated effects of the factors, the evidence provides a convinc- 
ing test of the importance of the institutional setting and implementation strategies. 

Institutional Setting and Organizational Climate 

Table 13 shows that the variables included under the category "Organizational 
Climate" had consistently significant effects on the success of a projeotras perceived 
by teachers. We will investigate the evidence for this finding in detai/ and consider 
the other institutional characteristics as control variables. 

All three columns of Table 13 show that the morale of teachers, at a school, the 
support of principal actors, and the teachers' willingness to put in extra effort 
significantly affected the percentage of goals achieved whether considered individq- 
ali^ or combined as overall indications o^ organizational climate. In short, innova- 
tive proje\.U were not "teacher-proof" t successful implen^ientation seemed to 
involve adaptation to the organizational tn/ironment and required commitment of 
participants and support from the school and the LEA. 

We defer ur.til Vol. Ill analyses and case studies of how organizational climate 
shapes adaptation of the project to its setting and thus determines its prospects for 
success. However, a sense of some of the systematic influences on the relationship 
of organizational clin^.ate to perceived successcan be gained by a variety of statistical 
comparisons. 

The positive coefficient of "element; ^ school" in Table 13 indicates that ele- 
mentary school projects were perceived a^ oignificantly more successful than junior 
or senior high school projects. Because the organizational characteristics of elemen- 
tary schools differ from those of junior and senior schools, it is reasonable to ask 
whether the effects of organizational climate on success were different for these 
different types of schools. Comparison of coL 1 with col. 2 of Table 13 shows that 
organizational climate mattered significantly for elementary schools, as it did for 
schools in general. Moreover, the differences in the coefficients of the support vari- 
ables for elementary school projects (as compared with all projects) appear to reflect 
^diner.ences between elementary and secondary schools. Namely, high school and 
junior high school teachers are usually members of relatively large and visible 
schools that are considerably more bureaucratized than elementary schools; thus, 
on one^b^nd, the superintendent may have mo* importance to the secondary school 
staffs, and, on the other hand, high schooHeachers mayl-equire mutual reinforce- 
ment from other teachers."^ For elementary school tfe^chers, the superintendent may 

, ^ It IS not untistiSTtO hear-elcmt'nlary school people, including adrfiinistrators, complain of a **high 
school bias" on the part of* school distnct^jfici^^^ 
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be remote, the teacher tends to be isolated in the classroom, and support from the 
principal may assume considerable importance for project implementation. 

To detect whether the importance of organizational climate (and other vari- 
ables as well) was limited to projects funded by one federal program as contrasted 
with others, the sample of TitL III projects was analyzed separately with the 
findings shown in col. 3 of Table 13. The pattern of significant coefficients shows tha* 
interpersonal relationships within the organization affected perceived success for 
Titlr III projects and. more generally, for projects in all the federal programs. 
Support from the superintendent seemed particularly important for Title III pro- 
jects. This result reflects the highly competitive, locally generated nature of the 
initiation and support stage . .he Title III program. Title III projects do not have 
their focus and funding opportunities already established and justified by federal 
program priorities; instead, they must seek the superintendent's support for goals, 
means, and priorities within the district; the evidence suggests that without such, 
support the chances of effective implementation were lessened. 

We can better understand the role of organizational cUmate, implementation 
'strategies, and other factors by analyzing and comparing their effect* for all four 
implementation outcomes. Table 14 presents the results of considering percentage 
of goals achieved, teacher change, difficulty of implementation, and implementation 
as laid out to be dependent on project characteristics, institutional setting, and 
federal programs. Tables 15 and 16 display, respectively, the results of parallel 
analyses for projects in elementary schools and. for projects funded by Title III. The 
patterns of significant coefficients across these tables demonstrate the importance 
of organizational climate, as shown by the following detailed discussion of particular 
effects of the independent variables comprising our measurement of organizational 
climate. 

Morale and Extra Effort. Focusing on the organizational climate variables 
for all projects in Table 14, we see that extra effort by the teacher not only increased 
perceived success but increased the amount of change in teachers' behavior in a 
major way (.26 is the largest single effect in the table). Moreover, extra effort on the 
teacher's part is strongly related to difficulty of implementation; that is, we infer 
teachers worked harder on more difficult projects and changed more. 

The findings on morale shown in Table 14 give further insight into these inter- 
personal dynamics. The higher the school's morale, the higher the project's percent- 
age of goals achieved and the higher the fidelity of the implementation to the initial 
plans. Yet higher morale also was associated with an ease of implementation and, 
most important, with a lack of teacher change. This seemingly counter-intUitive 
finding is not difficult to u.iderstand when one recognizes that change in a school 
setting hurts. 

As the case material will illustrate, it was not unusual-perhaps it is the norm 
—for "innovative" projects to have the ideology of change without the reality. In 
these instances, project participants often believed they were engaged in an interest- 
ing and useful departure from their standard practices. But because their implemen- 
tation strategies did not demand or elicit actual change in a teacher's relationship 
to the student, to other teachers, or to administrators, morale was higher and the 
project was implemented as initially laid out. Yet there were exceptional projects 
that demanded teacher cha.ige and adaptation of the organization to the project. 
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Table 16 
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Such change requirements aroused conflicts, lowered morale, but did, in fact, pro- 
voke teacher change. 

These relationships between morale and extra effort were perhaps even more 
pronounced for elementary schools, as Table 15 indicates. For Title III projects (see 
Table 16), the same tendencies exist in the data, but the findings are weaker. In 
particular, the effects of extra effort and morale on the fidelity of implementation 
are significantly lower. We interpret these results as reflecting programmatic char- 
acteristics of Title III. Namely, many locally conceived Title III projects had a 
tendency to submit proposals that were broad and diffuse; thus. Title III projects 
were more likely to deviate from their initial plans during implementation.® 

Support from Major Actors. The patterns of coefficients across Tables 14, 15, 
and 16 show that the principal was especially crucial to implementation. The varia- 
ble "Support from principal" had its strongest effects in elementary schools, where, 
of course, the administrator is a dominant figure. 

In particular, the quantitative results of Table 15, along with our field studies, 
suggest a hypothesis about the role played by principals in elementary schools. 
Insofar as they supported an innovation, they facilitated its implementation (thus, 
the coefficient of support from principal for implementation as laid out is a positively 
significant .17, and the coefficient Tor difficulty of implementation is a negatively 
significant -.13). The principals' opposition to projects sharply increased the pros- 
pects for failure, whereas their active or even passive support was necessary for 
perceived success (thus, the coefficient of support from principal for percentage of 
goals achieved is a positively significant .11). Yet the evidence of a weak and not 
significant effect on teacher change suggests that either (1) principals tended to 
support innovations that did not involve major change attempts or (2) their ability 
to influence staff' behavior was limited.® In short, the principal seemed to serve as 
either a facilitator or an inhibitor of change. 

Turning next to the superintendent's role. Table 14 shows the support from 
superintendents positively affected the fidelity of implementation and the percent- 
age of goals achieved but was negatively associated with teacher change and with 
difficulty of implementation. The strong negative effects of superintendents oc- 
curred primarily in elementary schools (compare Table 14 with Table 15). It may be 
the case that in the relatively few situations in which superintendents took a direct 
interest in elementary school projects, the projects were less difficult and less likely 
to produce change. However, a positive influence of superintendents occurred for 
Title III projects (compare Table 14 with Table 16). As we hypothesized earlier, 
superintendents may be more likely to concern themselves with, and have more 
commitment to, the locally initiated Title III projects. 

The different parts played by superintendents and principals during implemen- 
tation are not surprising in light of the organizational structure of school districts. 
The organizational remoteness of superintendents may mean they can provide only 
a generalized support that makes the school district receptive to innovations; such 
receptivity may be essential to the implementation of Title III projects. Elementary 
school principals appear to be "gatekeepers'' of change, either facili' Hing or inhibit- 

* Sec Sec III for an extended discussion of this point. 

» Several empirical studio of school systems have observed the limited influence of principals and 
other administratora on the behavior of teachers in their classroom. 
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ing implementation. Teachers themselves seem "isolated" in their classrooms, and 
alterations in their behavior may result less from the authority or influence of 
administrators than from their own commitment to change. 

Project Characteristics 

Of the three components comprising project characteristics, Tables 14 through 
16 indicate that implementation strategies— the explicit or implicit design choices 
made to implement the project— had the most consistently significant effects on 
implementation outcomes. We will next examine the detailed results for implemen- 
tation strategies with brief discussions of the eflects of project resources and educa- 
tional methods. 

Implementation Strategies, Almost independently of the educational tech- 
nology or method involved, implementers of mnovations have considerable freedom 
in putting their projects into practice. The choices listed in Tables 13 through 16 (and 
discussed iiT'^Sec. II> reflect, though do not exhaust, the major strategies suggested 
by the literature and experience. Each implemented project • ?velops its own strate- 
gic mixture. Thus, to investigate an individual project, it is a^ jropriate to observe 
its characteristic implementation strategy or syndrome. Howi ^r, for statistical 
purposes, we need to analyze the strategic elements separately. The following 
material discusses the effects on implementation outcome-? nf each strategic choice 
in turn but defers synthesizing the results until the conciu.^ion. 

Planning. Beginning with the first listed element of implementation strate- 
gies, the percentage of the project's budget spent in planning and project design was 
not significantly related to implementation outcomes in Tables 14 through 16. Pro- 
jects varied considerably in the amount of planning they did, and it would appear 
that the extremes of virtually no planning and of almost all planning in the first year 
or two were not characteristic of effectively implemented projects. But, in any 
event, the amount of planning may be less important than whether the quality of 
planning matched the needs of the project. That is, planning can assume different 
models of change (as discussed earlier, a technological, a rational, or an adaptive 
model). Unless the implicit planning model is congruent with the realities of project 
implementation, we hypothesize that teacher change and other goals will not be 
advanced. 

Staff Training. The more staff training (i.e., the time project teachers spent in 
training) the more likely was teacher change, particularly in Title III projects (see 
Tables 14 through 16). But the amount of staff training by itself did not significantly 
increase perceived success and tended toward decreasing success in elementary 
school innovations (see Table 15). 

Our fieldwork experience suggests an explanation for this discrepancy between 
change and perceived success. Many projects, including staff development projects 
funded by Title ill, do not seem to have linked training in new methods to appli- 
cation in the classroom. For example, pullout or pre-service trai.^ing may not have 
been able to anticipate day-to-day activities during implementation. More impor- 
tant, stafi' training that is not integrated with other strategic components that 
reinforce the teacher's attempts to implement a newly learned approach may have 

*^ This suggests that a nonlinear relationship might be used in future analysis. 
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little lasting efTect. We will return to this hypothesis when we consider the interac- 
tive effect of training along with staff meetings. 

Meetings. The positive significance of staff meetings for all the implementation 
outcomes, for all levelsjof schools, and for projects on all federal programs is clearly 
shown by Tables 14 through 16.*^ This finding is particularly important in light of 
the strong dependence of implementation on organizational climate. Regular and 
frequent meetings can facilitate communications and coordination and also enhance 
morale and the teachers' commitment to the projects, when teachers believe that 
the meetings contribute to the essential activities of the project. 

Meetings-Training Interaction. The value of meetings operating in conjunction 
with other elements of implementation strategy can be examined statistically by 
analyzing "interactions." In particular, preliminary analysis shows that although 
the amount of staff training time by itself is not significant for perceived success or 
fidelity of implementation, its interaction with meetings increases both perceived 
success and fidelity, as well as marginally enhancing teaciier change (see footnote 
a to Table 14). This result reinforces the hypothesis suggested earlier: Pre-service 
training, including technical assistance by consultants and outsiders, may be less 
effective than in-service and ongoing training linked to regular and frequent meet- 
ings of the project staff. 

Participation. Considering the emphasis we have placed on involvement and 
participation, the next finding is to be expected: If teachers felt they did not partici- 
pate in day-to-day decisions as the project was implemented, implementation v^as 
more difficult, and the chances of success, of fidelity to the project design, and of 
teacher chan£:e were reduced (see Tables 14 through 16 under the variable 'Teacher 
not participating in implementation decision"). This result was particularly strong 
for Title III projects, perhaps because of their local initiation. 

Flexibility. Project design flexibility would seem a priori to be a desirable trait. 
We operational ized this flexibility (in gross terms) by measuring the freedom of 
project directors, principals, and teachers to alter the project design. None of the 
measures produced significant and stable results, although teacher flexibility had 
somewhat stronger effects than the others and is therefore shown in the tables. This 
statistical result may be due to measurement error; or it may be that flexibility did 
not affect implementation outcomes; or it may be that flexibility should not be 
considered by itself (i.e., there is a specification error) but rather must be treated as 
part of complex leadership relationships on the project. We were unable to measure 
such leadership characteristics in the first year of the study. Project leadership and 
flexibility remain areas in which additional research is needed. 

Local Material Development. The development of materials locally by project 
participants is shown by the case studies to have a considerable effect on implemen- 
tation. Unfortunately, our measurement of this variable is simply v^hether a project 
did or did not develop its own materials (as reported by the project director). This 
measurement is contaminated jy the tendency of projects to "reinvent the wheel;" 
that is, most projects tend to adapt ^ven prepackaged material to their own setti' g. 
Consequently, our operational measure iS probably more accurate for projects that 

' » The operational variable used in the analysis is the value of meetings as reported by teachers This 
variable is highly correlated (.7i) with the frequency of meetings 

This variable is a dummy variable, with "Teacher not participating" coded as 1 and "Teacher 
participating" coded as 0 ^ 
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accepted prepackaging in toto and, in this sense, did not develop their own materials. 
Given these definitions, not developing local materials (i.e., using prepackaged 
materials without adaptation) was not significantly related to implementation out- 
comes, except for Title III projects where it decreased perceived success (see Tables 
14 through 16). A fuller discussion, of this subject is deferred until Vol. III. 

Projects Located in Special Units. Projects that were primarily located in 
special units outside of classrooms tenaed to be ''puUout" projects in which either 
selected students or a whole class spent one class period, once a week, in the project's 
laboratory or resource center. Although the statistical data are not significant, the 
effects of the puUout projects seem to depend on whether they were intended for 
remedial purposes (e.g., in reading or math in elementary schools) or for enrichment 
(e.g., career education often in junior or senior high schools). The enrichment units ^ 
tended to be implemented as laid out and positively affected the percentage of goals 
achieved; the remedial laboratory was often difficult to implement. In either case, 
projects not located in the classroom usually did not engage the teacher in the 
project and were thus less likely to result in changes in teachers' behavior (see Table 
15). 

Project Coverage. Of the 104 projects in the Rand sample that served high 
schools, 65 also served elementary schools.*^ These broad projects included some 
Title III, Right-To-Read, and Vocational Education projects. Such broad project 
coverage had a strong negative effect on the effective implementation of projects, 
particularly those funded by Title III (see Tables 13 and 16). 

These projects seemed to have had severe management and administrative 
problems resulting from the attempt to integrate similar goals and treatments 
across different types of schools. For example, some reading projects spanning school 
levels promoted complex and ambitious plans for diagnostic and prescriptive meth- 
ods. Although some of these projects did produce significant change ' teacher 
behavior at the elementary level, where teachers view reading as one of the* central 
teaching tasks (see Table 15), they may not have gained the necessary commitment 
of high school teachers. The net result was the apparent failure of the project to 
realize its high expectations. 

Change or Effort Required of Teachers. In addition to staff training and meet- 
ings, the implementation strategies most likely to increase teacher change involved 
those requiring change. Tables 14 through 16 indicate that this important finding 

The dispersion of projects in the Rand sample for the different school levels was 

Number of Projects 



Types of Schools Served ni Rand Sample 

Both elf^mentary and high schools, or both 

elementary and junior h'l^h *. . 97 

F)lementary schools, but no high schools, no 

junior highs, and no out-of-school or 

a<lult programs 106 

High schools and/or junior highs, but no 

elementary schools or preschools 57 

Other combinations of school types, including, 

for example, exclusively nonpublic or 

exclusively out-of-school and adult 

programs 18 

Total . 278 
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held whether the required change involved a specific teaching technique (eg., on a 
Right-To-Read project) or an overall change in teaching style (e.g., in an open- 
classroom project). Projects requiring teacher change and extra effort were difficult 
to implement and generally did not adhere to the initial project design. Perhaps 
because of their ambitious nature, they tended— although the estimates are not 
significant— to have a negative effect on the achievement of project goals. 

Mutual Adaptation Strategy. Thus far we have discussed the findings for the 
effects of implementation strategies item by item.** However, each project employed 
its own combination of strategic choices that defined its particular implementation 
strategy. Although we cannot statistically analyze individual combinations, the data 
do suggest characteristics of an implementation strategy that might be more likely 
to result in high levels of teacher change and achievement of project goals. In 
particular, uequent and regular staff meetings, staff training held in conjunction 
with meetings, and project requirements placed on teachers to alter their behavior 
appear to be elements that worked together so that project participants could adapt 
to the project and vice versa. This combination seemed to comprise key components 
of an implementation strategy that might be called a mutual adaptation strategy. 
We hypothesize that a mutual adaptation strategy may be characteristic of projects 
that result in significant change. This hypothesis will be further investigated in the 
case studies of Vol. III. 

Project Resources. The resources of projects were measured in terms of (1) 
the absolute level of funding in the year of the maximum grant from all sources, (2) 
the number of students served by the project, and (3) the funding per project student. 
Aside from some exceptions to be noted. Tables 14 through 16 show that these 
measures did not have strong effects on implementation outcomes. 

The apparent irrelevance of project resources for most implementation out- 
comes may be the result of the way the vanables were measured. For example, 
rather than using the maximum per year grant, the average per year grant might 
be used. (See Sec. II for a discussion.) Or rather than the linear specification of the 
variables, a more complicated expression might better capture the effect of re- 
sources. For example, it could be argued that low levels of funding f very high levels 
of funding diminish the chances for effective implementation, but that projects in 
the range of $100,000 to $150,000 have increased chances of being effective. Or it 
may be that, within the range of grant variation involved in the federal demonstra- 

'* As indicated previously, the list of strategies is not exhaustive. Some possible strategies (e.g., how 
students should be approached in order to implement a project) were not measured. Other strategies were 
meastired but could not be incorporated into the analysis for technical or measurement reasons For 
example, one implementation strategy involved incentives offered to the project staff. The survey did not 
measure such incentives as professional rewards or increased status, which seem effective in motivating 
commitment (see Vol. III). We did gather data on whether the staff was paid in part or in full for training 
on the project. Preliminary analysis indicates that these money incentives had no significant influence 
on the effective implementation of a project or on changes in teacher behavior The variables for 
paid in part or in full were not entered into the multivariate analysis shown in Tables 13 through 16 
because of multicollinearity problems. However, numerous multiple regressions not detailed m this 
report support the above conclusion, as do the following pairwise correlations: 

Percentage of Teacher 
Goals Achieved Change 

Trainmg paid for in part .... . . 03 ~.00 

Training paid for in full -05 .02 



Number of observations = 509. 
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tion projects, differences in funding did in fact have little effect. In lieu of further 
analysis, the latter hypothesis should be seriously entertained, namely, that varia- 
tions in funding above a minimum level did not affect implementati(jn outcomes. 

Large target-group size projects did significantly reduce the chances for teacher 
change, particularly in elementary schools (see Table 15). In addition, the more 
concentrated the funding was on Title III projects, the more likely was teacher 
change. (See Table 16.) Finally, Title III projects that had high levels-ef funding and 
served large numbers of students — a combination implying high initial investment 
in such capital expenditures as computers or school wide resource centers — were 
.likely to be implemented as laid out, perhaps because little change of the organi- 
zation was required, and thus adaptation of the project did hot occur. 

Educational Methods. Earlier in this section we examined the effects on 
implementation outcomes of variation in educational methods within each federal 
change agent program. We found that differences in the level of the methods ex- 
plained only a small percentage of the variation in implementation outcomes. The 
analyses shown in Tables 14 through 16 test a similar hypothesis but with the 
addition of controls for implementation strategies, project resources, and institu- 
tional setting variables, Again^ variation in the level of educational methods as 
measured by factor analysis scores did not strongly affect implementation outcomes. 
In addition to measurement errors, this finding may be faulty because of a statistical 
artifact. Because educational methods were weakly linked with other design choices 
and with federal programs, the estimates of their separate effects mav be somewhat 
inaccurate (see App. B). Even if these problems were more significant than we 
believe they are, there seems little doubt that differences in the educational method 
or technology cannot account for the variation in implementation outcomes by itself, 
that is, without reference to the institutional setting and implementation strategies. 



FEDERAL PROGRAMS 

For all the reasons mentioned earlier in this section, we must be cautious in 
weighing the net effect on implementation outcomes of one federal program relative 
to others. We have analyzed the effect on implementation outcomes of federal 
change agent programs by themselves and then in conjunction with variations in 
the level of educational methods within programs. Although some useful distinc- 
tions were drawn, the differences between the programs explained little of the 
variation in implementation outcomes. 

The analysis involved in Tables 14 through 16 takes the next step of introduc- 
ing, and thus controlling for, project and institutional characteristics. The results 
indicate that, other things being equal, Right-To-Read projects and Bilingual, Title 
VII projects were less likely to produce teacher change than Title III projects, and 
that Bilingual, Title VII projects were relatively difficult to implement. However, 
perhaps the most important point suggested by the tables is how little the knowledge 
of federal sponsorship helps in predicting project implementation outcomes. In 
short, the variation between programs was less important than the variation be- 
tween projects' institutional setting and implementation strategies. 

so 
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SUMMARY OF MAJOR FINDINGS AND HYPOTHESES 

L The effective implementation of innovative projects was primariiy depen- 
dent on a supportive institutional setting and on an implementation strate- 
gy that fostered the mutual adaptation of the staff'to the project's demands, 
and of the project^s design to the reality of its setting. 

2. Projects funded by the same federal program showed considerable varia- 
tion in their implementation strategies and institutional settings. These 
within-program variations affected project implementation more signifi- 
cantly than did the differences between federal programs. 

3. Projects using similar educational methods or technologies varied consid- 
erably in their implementation strategies and institutional settings. The 
variations affected project implementation more significantly than did the 
differences between the educational methods or technologies themselves^ 

4. Superintendents, who.tend to be organizationally remote, provided a gen- 
eralized support that may have made schools receptive to innovations; such 
receptivity may be essential to Title III projects. Elementary school princi- 
pals appear to have been "gatekeepers" of change, either facilitating or 
inhibiting implementation. 

5. The following elements of implementation strategies promoted teacher 

change: 

a. Staff* training. 

b. Frequent and regular meetings. 

c. Staff meetings held in conjunction with staff training. 

d. The quality and amount of change required by the project. 

The following elements of implementation strategies inhibited perceived 
success or teacher change: 

a. The lack of the above elements. 

b. Teachers not participating in day-to-day implementation decisions. 

c. For Title III projects," the lacjc of local material development. 

The following elements of implementation strategies were not significant- 
ly related to implementation outcomes: 

a. The quanfi^y of planning. 

b. Participants* freedom to alte^* the basic project design on difficult pro- 
jects. 

c. Part or full pay for training, 

6. Elementary school projects were more effectively implemented than jun- 
ior or senior high school projects and were more likely to produce teacher 
change. 

7. Within the range of variation of differential funding considered here, a 
project's funding level did not have significant effects on teacher change or 
perceived success. Projects that serve most of, or the entire, student body 
of elementary schools were unlikely to produce teacher change. The more 
concentrated were the resources of Title III projects, the more likely was 
teacher change. P,'^^ 



V. EXPECTED CONTINUATION: DISTRICT LEVEL 



The federal demonstration projects provide "seed money" to local school dis- 
tricts in the hopes that such extra funds will stimulate the adoption and spread of 
innovative practices. In the first year of this study, we examined the "trial" period 
of the innovation during which the project was initiated and implemented with 
financing, at least in part, by federal funds. But a major test of the impact of these 
funds is whether successful projects can take root after the termination of federal 
funds, that is, whether successful innovations are continued by the LEA and result 
in enduring changes in local educational practices. 

Continuation is a difficult question to assess and analyze. Indeed, our first-year 
research suggests that the effects of educational innovations need to be measured 
in a variety of ways for different levels of the school organization. For example, 
innovative projects can produce change in teachers' classroom behavior and activi- 
ties, can affect the standard operating practices and educational methods within 
schools, and can alter districtwide priorities and procedures. Because of these di- 
verse and complex possibilities, it is difficult to evaluate the extent to which these 
changes are continued aflertniTces^^^^ Moreover,'tTie^vario^^ 
outcomas^may be "continued" in partial and unanticipated ways. Thus, innovative 
classrooni practices can bie continued by an individual teacher independently of the 
formal ^bntinuance of the project or, indeed of the awareness of school officials. Such 
changes can be both enduring and sig^^ifi^ nt, and thus need to be weighed as part 
of the project's long-run effect. 

Continuation will be the focus of the second phase of Rand's.overall research. 
However, during the 1973-74 survey and case studies of projects in their last year 
of federal funding, we did gather information about expected contin lation. This 
information allowed us to conduct preliminary analyses of continuation and to 
formulate hypotheses for the second phase in which we will observe projects after 
federal funding has ended. Volume III examines the incorporatit)j:i of project ele- 
ments at the school or individual level. This section analyzes the district continua- 
tion decision.^ 

We expect the project's success or failure during its implementation stage to 
affect a LEA's decision about continuation. But political and economic factors as well 
as internal organizational pressures and constraints may also influence the choices 
made about the project^s future. This section explores some of these complex rela- 
tionships in order to estaBnsh preliminary findings and working hypotheses fpr 
additional research. 

We describe below the survey items used to measure continuation and project 

' Several studies of Title HI contnuation were used as a biisis for many of the hypotheses investigated 
for the various federal programs Although this study has a different focus, some of the results are 
comparable. Among these studies are Jerome B Bnghtman, "The Continuation Rate of Three-Year 
KSKA Title HI Projects," A Report to the President's National AdiUiory Council December 15, 1971: 
Norman E. Hearn. "Innovative Educational Programs A Study of the Influence of Selected Variables 
upon Their Continuation Following the Termination of Three- Year l^EA Title fll Grants," Ph.D. diss , 
George Wiishlngton Univeriiity, Washington, D C . 1969, and Anthony N Polemeni, ''A^tudy of Title III 
Projects afler the Approved Funding Periods," Ph D. diss., School of Education. St. John's University, 
Jamaica. New York, 1969 0'*^> 
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outcomes at the district level. Then we compare these outcomes for the various 
federal change agent programs to determine whether the variation in the expected 
continuation of projects was related to the differences between the programs. Fol- 
lowing these program comparisons, we present a.^J tert explicit hypotheses relating 
political, economic, and organizational factors to continuation. 



MEASURES OF PROJECT OU • ' v AT DISTRICT LEVEL 

We previously argued that implementation outcomes at the clas' , om level 
were best measured by the responses of teachers to a series of questions. However, 
the continuation decision is primarily a district decision, and studies have found that 
the superintendent is the principal actor ii. making this decision.^ Therefore, we 
relied on superintendents' answers as the best measure of expected continuation ^ 

To complement the analysis of continuation, we also used measures of the 
superintendents' perception of project succe nd of the MfTiculty of implementing 
the project.^ The reader should be warnec' ' i superintend^^ntV responses In 
these areas correlated weakly with those o» :tors, perhaps because s?:perin. 

tendents are concerned with broader priori i^nce, the superintendents' vi-ws 

' See Bnghtman 11971) and Hearn (1969). 

' Although the survey asked a number of questions, responses to the folio^A'ing question will be used 
in the analysis: 

Do you expect this project will be expanded continued at the present level cut back, or not 
i'Ontinued at all^fter termination of federal fumh? 

Expanded ... ... - 25.39%^ 

Continued at the present >^vel 26.45% 

Cui hack ........ 40 74 % ^ 

Not continued at all • 7.40% 

Number of answers = 220. 

* The specific questions and frequency of responses were 

Overall when you take into account the goals the projeO sta-\ ' Jith and the resources it had, 
what p'ViA)rtion of its goals would you say the project achiei ' ' 

Mean = 55 7 % 
Median = 65% 
Standard deviation = 33% 

Overall how difficult do you feel that this project has been to implement, very difficult, r.xoder- 
ately difficult, a litth difficult, or not at all difficult'^ 

Very difficult 

Moderately difficult . 
A little 'ifficult . . . 
Not at all difficult . 

Number of ancrt^ers = 221. 

* Averaging the answers of teachers on a project (maximum was four teachers at two schools in the 
Rand sample) and averagmg the answers ofprmcipals on a project (maximum was two), we obtained the 
pairwise correlator of the various outcome measures (including contm lation to be defined subsequent- 
Iv^ shown below for teachers, prmcipals, and superintendents? 

f3 



13.16%L 
44.74% 
28.42% 
13.68% 
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of success and fidelity of implementation need to be analyzed differently than Sec. 
IV's analysis o. teacher responses. 

In addition to the limitation on the analysis implied by the use of expected 
continuation, several other qualifications should be recognized. Some projects re- 
ported as likely"^ be continued may not be. Moreover, some projects that are 
initially continued may be phased out several years later.^ Therefore, our figures for 
continuation using the measure of expected continuation are likely to be inflated 
However, given sample limitations, the relative, rather than the absolute, level of 
conlinuation is our primary concern.^ 

Another limitation was discovered during the pre-testing of the survey in 
November 1973, We found- that superintendents, as well as other participants of 
projects in their last year of federal f^ncling, had begun to decide on continuation, 
but only in broad terms. They seemed fairly certain about the general question of 
continuation, but not about such important details as future budget allocations and 
future staffing levels. Therefore, the measurement of expected continuation was a 
gross measure, lacking the precision that the next phase of this study plans to 
attain.^ 



PROGRAM COMPARISONS ON CONTINUATION 

To what extent did the differences between federal change agent programs 
explain the variation in expected continuation and other project outcomes? To 
examine thi? question, we conducted a statistical analysis similar to the program 
comparisons done for implementation-classroom level outcomes presented in the 
preceding, section. 

Table 17 displays the results of compari. " the federal programs on their aver- 
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.26 
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.02 
-.06 
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-.08 
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-.04 
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Superintendent 
Dif. 
Goals 
Contin 


.2(5 
-.02 
-.00 


-.08 
.1 1 
.10 


-.23 
.3o 
19 


-.13 

.31 ' 
31 ^ 


- 19 
.35 
.20 


-.00 
• ".00 
.1 1 


.06 
-.0.1 
.18 


-10 
.38 

.25 


.02 


15 



"* See Bnghtman (1971) and his computation of a "real'* continuation rate 
^ We exfx^ct relative levels to he less affected by these overestimates than absolut^levels Nonethele^.., 
given the above-mentioned problems, the result^J must be treated cautiously • 

" For the sake of statistical analysis, we will use the van ible asS if it were continuously* rather than 
categorically, measured with 1 noi continued, 2^ cut bad , 3 Vontinued at present level, and 4 
expanded. ^ /i 
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age outcomes for the projects in thp sample. For the sake of comparison, the teach- 
ers' average responses on percentage of goals achieved and difficulty of implementa- 
tion K-e presented next to the superintendents' average answers. Discounting the 
absolute values of these figures, the data suggest a general observation about the « 
relative results: There was comparatively little difference between the programs 
in the average outcomes of their project* measured in the above terms. This finding 
parallels similar results Tor implementation outcomes. Again we suspect that pro- 
gram differences cannot be adequately assessed without taking into account such 
elements as the educ Jtional methods employed on a project or its funding level or 
the size of the schooi district. 

Before introducing some of these elements, Table 17 reveals several marginal^ 
differences between tKe federal programs that illuminate the differential effects of 
their distinctive managemen."; strategies: 

1. Vocational Education projects were the least likely to be continued even 
though superintendents viewed them as accomplishing a high percentage 
of their goals. This finding is consistent with the'inferences we drew from 
the classroom-level data, namely, that Vocational Education projects 
seemed to achieve a high percentage of their goals because they were not 
attempting major innovations but simply added 'Career enrichment materi- 
als to the curriculum. The superintendent data ouggest that these add-on 

* ' projects were .tl^e lea'st likely to be continued by the district when the 
federal funds are 'terminated.® 

2. Bilingual, Title VII projects were viewed as the most difficult to implement 
and as having achieved the lowest percentage of their goals. Nonetheless, 
their expected continuation was, not significantly lower than the average 
of projects funded by the other programs. (Subsequent analysis shows that 

"'^ Bilingual projects tended to have high expected continuation, once other 
factors are controlled.) Our field studies -suggest an explanation for this 
disparity between the uneven past performance of Title VII projects during 
the trial period and their likely co.'.tinuatton: Ucal constituencies have 
I mobilized support or created demand for bilingual education within many 
school districts and state agencies and thus produced pressure for a local 
and state commitment to deal with their problems.'" 

3. Superintendents reported Right-To-Read projects as relatively the easiest 
to implement and somewhat. mOre likely to be continued than Vocational 

' Education projects. Subsequent analysis suggests that projects with educa- 
tional goals that^re more tentral or important to district educational 
concerns are^ more likely to be continued. Thus, Right-To-Read projects 
concerned with the core task of reauing may have generated more commit- 
ment taan enrichment-type Vocational Education projects." 

t 

' It IS interesting to note that teachers found the Voc<itional Fducation projects markedly less diflicult 
to implement than.the superintendents reported. Could it be that superln^ndents felt the projects were- 
easier than they cared to admit? * ^ ■ . c ■ j 

'» Note the differences l)etween the teachers' and the superintendent^' assessments, Supermtendenta 
appeared to accord le»s success to Bilingual projects, perhaps because they are more concerned wuh 
broader political issues. or.- >^ m iv j 

" Note t'he discrepancy between the teachers' PQrception of the difficulty of Right- Fo-Read projects 
and the superintendents' view that these projects were the easiest to implement From the stahdpoint 
of hich-level administrators, the packaged management alternatives and the noncompetitiveness of the 
Right-To-Read grants might have been quite attractive; unfortunately, these advantages Were oOen 
dysfunctional at the classroom level ^ ^ 
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4. The results for Title III projects are difficult to interpret Superintendents 
viewed them as more difficult to implement than Right-To*Read projects, 
as more successful than Title VII projects, and as more likely to be con- 
tinued than Vocational Education projects Various explanations of these 
gross comparisons could be offered. But considering the diversity of educa- 
tional methods and activities encompassed by Title III, interpretations 
would be more accurate afler additional elements are introduced into the 
analysis. 

We can elaborate these general findings by analyzing the effects of variation in 
educational methods within each federal change agent prograri Table 18 presents 
the results of a multiple regression analysis designed to examine these etfects.^^ 

The introduction of within-program variations helps to explain somewhat more 
of the variation in project outcomes but not much more. Nonetheless, several infor- 
mative findings do emerge. 



See Sec IV for a discussion of the statistK ' strui,ture oi the parallel analysis for implenientfition 
outcomes c 



Table 18 

t'R(X,RAM-HMHUD LC^PARlSONb ON CONTINUATION: SUPERI?rrENDENTS ' RESPONSES 



Educational Methods 
bv Prograrj 



Expected 
Continuation 



Re^iresslon CoeffUlent3 and Standard Errors 



Percent- 
age of 
GoaU 

Achieved 



Difficulty of 
ImpleDencatlon 



Title III 
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Classrooni organization 
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staff Ing 
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Right-To-Read 

Behavioral modification 
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Intensive traditional 
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-.2 
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2.67 
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.21 
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(.07) 

.05 
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2.12 
(.40) 
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NOTE- An aste'lsk (*) indicate* significance at the .15 level, a 
d-uble asterisk (**) Indicates significance at the .10 level, and a 
tr.Mc asterisk (***) Indicates significance .-.t the .01 level. Coef- 
ficients of progratas are, of course, all alf.nlf Icantly dlffe -nt frot., 
zero, but a aore rclevan; cooparlaon U with the average of projects 
when the variation of the level ot educational r^ethod. Is Ignored. 
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1 Title III projects with high levels of c!ass/oom organization changes were 
more likely to be continued than other Title III projects, even though they 
vjere viewed as more difficult to implement and no more successful. This 
finding suggests c hypothesis to be tested shortly — classroom organization 
change projects were likely to have had the commitment of the district and 
thus were likely to be continued independently uf their perceived success 
during implementation. 

2. The superintendents* attitudes toward the continuation Oi Right-To-Read 
projects may have depended more on the reading focus and programmatic 
characteristics of Right-To-Read than on the specific level of educational 
methods employed on individual projects. The same result— and inference 
—held for Vocational Education (with the exception of classroom organi- 
zatioii changes, which were seldom used in Vocational Education projects). 

3 The expected continuation and perceived success of Bilingual projects de- 
pended on the mix of methods employed. High levels of behavioral modifi- 
cation instruments and of classroom organization changes were seen as 
difficult, relatively unsuccessful, and less likely to be continued. In con- 
trast, intensive use of traditional staffing and enrichment tended to con- 
tribute to perceived success and to expected continuation. 

In sum, despite some important marginal effects, neither the type of federal 
change agent program nor the type and level of use of educational methods greatly 
influence expected continuation. Next we will identify other factors that might 
affect continuation. 



IDENTIFYING POSSIBLE FACTORS AFFECTING EXPECTED 
CONTINUATION 

The last stage of the innovative process is incorporation, the process by which 
the practices, activities, or behavior developed during project implementation 
become part of the standard procedures in the classroom, school, or district. Un- 
doubtedly, the most immediate sense of incorporation occurs in the changes in 
teachers* behavior brought about by participation in an innovative project. For 
example, a teacher involved in classroom organization methods may come to assimi 
late individualized instruction practicesjSuch individual change lasts beyond the 
life of any special project. 

However, individual teacher change in school settings tends to be isolated. 
Teachers, particularly in elementary schools, have little influence on administrative 
decisions that structure the school and its educational and bureaucratic practices; 
yet they operate behind "closed doors'' and thus have considerable freedom to follow 
their own teaching practices. Thi^ "culture of schools*' inhibits the diffusion of 
individual change from one teacher to another in the same school and certainly 
across schooljines or district boundaries. 

Consequently, we would not expect projects concerned with the development of 
staff to stimulate additional change beyond their original location unless schocl 
administrators and LEA officials supported the innovative project. 

In the political-economic context of the LEA, the test of support generally comes 
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when the funding for special projects is about to run out. It is then that a decision 
must be made as to whether the project in some form or anoti^er should be continued. 

This is not to say that continuation is solely determined by economic considera- 
tions. Resources matter, of course. But w.thin the same overall budget, districts can 
juggle their priorities. The importance of the funding decision on continuation 
reflects a political-ortjanizational reality: Administrative decisions tend to be timed 
on budget planning cycles. The end of the temporary federal funding for an innova- 
tive project forces the LEA to assess its priorities— forces a decision about continua- 
tion. 

A simple conceptualization of the nature of this decision will help identify 
factors that might affect continuation. We conceive of the decision to continue an 
innovation as a function of four general considerations— the likelihood of its future 
success, the importance of the educational need served by the project, the resources 
required by the project relative to district resources, and the organizational-political 
forces inhibiting or promoting the innovation. 

Each of these theoretically important considerations was measured using sur- 
vey and census data. The remainder of this section defines their meaning and 
discusses the variables used as their operational indicators for statistical purposes. 
To give the reader a sense of the tendencies in the data. Table 19 -lisplays pairwise 
correlations between expected continuation and these variables. Pairwise correla- 
tions can be misleading^ however, because of multiple interrelationships among 
variables. Therefore, v/e will next analyze multivariate relationships between these 
variables and continuation. 

Estimated Success of an Innovation 

In deciding on the future prospects of an innovation, superintendents may draw 
on information about the innovation s implementation during the trial period of 
federal funding. One source of such information might be evaluation reports. How- 
ever, our fieldwork experience suggests that superintendents may not pay much 
attention to evaluation reports. They seem instead to rely on formal and informal 
discussions with project participants. We used this insight about the communica- 
tions network in school systems to develop a measure of the superintendent's percep- 
tion of a project's success that was uncontaminated by his opinion of the importance 
of the project.*^ 

In particular, we assumed that the superintendent s perception of success (mea- 
sured by his report on the percentage of the project's goals achieved) depended on 
the principals' and teachers' perception of implementation o. tcomes. Because more 
than one principal, as well as several teachers, could have been involved in an 
innovative project, we assumed that the superintendent subcmsciously integrated 
the responses of these different individuals— who not infrequently disagreed--by 
averaging the principals' responses, averaging the teachers' responses, and 'A'eigh- 

The superintendent's answer to a question about the succejis of a proj(»ct may be inextricably 
related to his view of the project s importance (as well as to his own characteristics) Therefore, we would 
introduce statistical errors (a simultaneity bias) into tlie analysis if we were <o upe both variables as 
explanatory variables for a project's continuation Instead, our "measurement" of project success is an 
estimate of the superintendent's answer about success, one that is based on variables, that can he assumed 
to be unrelated (or exogenous* to the superintendent's perception of importance or hi.» personal character 
istics 
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Table 19 

PAIRWISE CORRELATIONS OF VARIABLES WITH EXPECTED CONTn^UAnON 



Explanatory Factors 



All 
Projects 



Project Considerations 

Perceived success (superintendent) 

Perceived difficulty (superintendent) 

Centrality 

Funding level 

Nuniber of students served 

Funding per student 

Project covers high and elenentary 

schools 
Project is special unit 

Percentage of project budget for planning 
Project not locally developed 



.15 
.02 
.15 
-.09 
-.02 
-.13 

-.23 
-.07 
.15 
-.07 



Institutional Characteristics 

/School district enrollment (log) 
Percentage of population black 
Percentage of 
Percentage of 
income j 
Setting i Percentage'of 



population Spanish 
population with $25,000 



population poor 
SMSA^ (rural versus urban) 
Local expenditure per pupil 
State expenditure per pupil 
Adequacy of finances (superintendent's 
view) 

Extent of social-political dif f icultie" 
Difficulty from test scores 
Difficulty from community 
Community supports innovations 
Ccmmunity criticizes innovations 
School board consensus [ 

(Initiate projects if additional! resources 
Initiate projects if can be continued 
Propensity to adopt innovatioas 
Project supported by prlncipa 
Superintendent * s tenure 



External 
forces 



acteristics 



.24 
.12 
.14 

.11 
.11 

.06 



.18 
.18 
.11 



.05 
.14 
>29 
.32 
.25 
.22 



Educational Methods 

Behavioral modification 
Enrichment 

Classroom organizational change 
Intensive traditional staffing 
Administrative change 



.10 
.15 
.23 
.05 
.12 



Federal Programs 



Right-To-Read 
Bilingual^ Title VII 
Federal Vocational Education 
State Vocational Education 
Title III 



.05 
-.04 
-.02 
-.15 

.10 



Number of observations 



122 



Standard Metropolitan Statiatical^Area. 

00 
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ing the disagreement between people at the same level.** We call the resulting 
measure the estimated success of the project as perceived by the superintendent. 

Educational Needs 

Innovative projects are designed to serve educational needs and priorities with- 
in the district. A cynical view of school district decisionmaking holds that orojects 
are continued or not regardless of their educational value; a naive view places full 
weight on educational value. The facts undoubtedly lie somewhere between these 
extremes. To estimate the importance of educational needs for a project's continua- 
tion, we used the concept of centrality (see Vol. I) or how close tlie goals of a project 
are to the major educational objectives of the district.*^ For example, a superinten- 
dent may consider a remedial math project to be more central than a project com- 
prised of field trips to the zoo. (See Table 19 for the bivariate correlation between 
centrality and expected continuation.) 

Resources 

It is a truism to say that a district cannot continue a project unless it can afford 
to do so. Schools are always limited in their resources. Yet within budgetary limits 
they <^an manipulate priorities. Moreover, some innovations are aimed at replacing 
existing activities, end thus their incremental cost to the district may be small 
relative to their accounting cost. The relationship between the absolute cost of a 
project and its continuation is an important but unresolved issue, one we will be 
examining subsequently. 

The ideal measure of cost would involve computations of project incremental 
cost relative to the district's discretionary or slack funds. Unfortunately, we could 

^* Statistically, we predicted the superintendent's perceived success by estimating the following 
equation- 
Superintendent: Percentage of Goals Achieved = 
a + (ave. principal % goals achieved) + bg (ave. principal difficulty of impL) 
+ b^ (ave. principal impl. as laid ouc) + (var. principal % goals achieved) 
+ (var. principal difficulty of impl.) + b^ (var. principal impl. as laid out) 
+ b^ (ave. teacher % goals achieved) + b^^^ (ave. teacher change) 
+ bg (ave. teacher difficulty of impl.) + b^Q (ave. teacher impl. as laid out) 
+ b^j (teacher variance % goals achieved) 
+ bj2 (teacher variance teacher change) 
+ bjg (teacher variance difficulty of impl.) 
+ bj^ (teacher variance impl. as laid out) 

This equation forms the first stage of a twc>^ti»ge least-squares estimation in which the above indepen- 
dent variables are instruments and the superintendent's perceived success is endogenous. 

** The specific question asked of superintendents and the frequency of their .-esponses were: 
How would you rate this prcject in terms of how close \ts goals a/t to the major educational 
objectives of this district^ Would you say very close, moderately clr^e, or not very closed 

Very close 79% 

Moderately close '20% 

Not very close 1% 



Number of answers 186. 
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not measure these elusive quantities. As surrogates, we have used, in preliminary 
analysis, dc« a on 

1. The project resources: its level of funding in its biggest year, the funding 
level relative to the number of students served by the project, and the 
spread of the project over grade levels. 

2. Demographic characteristics associated with (among other factors) eco- 
nomic slack or the lack of it* district size, i>ercentage of minorities in the 
district, percentage of wealthy and percentage of poor in the district, and 
the district's urbannessy' 

3. Rough measures of diS-rict finances: local expenditure per pupil, state 
expenditure per pupil, and the adequacy of the district's financial situation^ 
as reported by the superintendent (see Table 19). 

Appendix C examines the efiectsof many of these variables on the district's propen- 
sity to innovate.^® 

Organizational-Political Forces 

Volume I argued that bureaucratic and political forces within school districts 
can either inhibit or promote innovations. In the instance of the continuation deci- 
sion, an innovative project necessarily creates a set of constituencies within, and 
occasionally outside of. the district. Participants ca.n become psychologically com- 
mitted to the project, moreover, they can come to see their self-interest involved in 
whether a project is continued. Detractors and supporters form de facto pressure 
groups. Although we could offer many specific hypotheses related to the importance 
of organizational-political forces, this section can measure few of them, and we thus 
limited the statistical analysis to a small number of significant eflfects. 

The survey asked superintendents about external political-social forces and 
about internal organizational characteristics. The measurements of external forces 
included the superintendent's assessment of the extent of social-political difficulties 
in the district, specific difficulties arising Trom student test scores and with the 
con.munity, the community's attitude toward innovations and its general tendency 
to riticize the district, and whether the school board was in gene'^al agreement or 
divvied over district goals. (See Table 19 for bivariate correlations with continua- 
tion.) 

The survey did not delve directly into internal organizational relationships— 
for example, the lot as of decisionmaking or the nature of authority relationships 
(see Vol. i >— but asked several questions that attempted to tap broad characteristics. 
One series of questions dealt with identifying which items from a list of 22 common 
educational innovations were adopted by the district The responses to these ques- 
tions were fashioned into a scale of the district's propensity to adopt innovations, 
which was used to test the following hypothesis: The higher the propensity of a 
district to adopt innovations, the more likely it will be to continue a project. The 
positive correlation of .32 shown in Table 19 tends to support this hypothesis, at least 
in the bivariate case.*' 

Although the analysis of the district s propensity to innovate? otft*^ several significant findings, it 
IS somewhat extraneous to the key Questions raised in this volume It was appended because of it« 
importance in selecting district characteristics for the above a,nalysis 

This hypothesis wiil not be further tested by multivariate means in thi.s volume because ofmulticol 
linearity problems For a det^iiled discussion of the district's propensity to innovat^^, see App C 

0.'> 
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We also asked questions about the initiation of special projects of the type 
funded by federal change agent programs. Based on earlier studies and on our 
fieldwork, we hypothesized that there are two general attitudes toward initiation: 
at one extreme, an opportunistic attitude in which special projects were treated as 
add-ons and would only be initiated if resources from outside the district were 
available (e.g., a zoo project); at the other extreme, a problem-solving attitude in 
which funding was sought for projects that were intended by the district to attack 
priority needs (e.g., remedial math) and that were thus slated for continuation before 
the project's implementation. The hypothesis to be tested is obvious— opportunistic 
cally initiated projects are unlikely to be continued, whereas projects initiated with 
the expectation of continuation are likely to be continued.*^ 

Considering the bureaucratic importance of principals in the operation of the 
school distVict, we asked project directors the extent to which principals, as well as 
other actor^, supported the project. We will test the hypothesis that such support 
is necessary to continuation. (Table 19 indicates a positive correlation of ,25.) 

The final variables related to continuation were superintendents* character- 
istics. Of a variety of personal characteristics, our preliminary analysis suggested 
that the superintendent's tenure had significant and positive effects for adoption of 
innovations (Possible reasons for this effect are discussed in App. C ) We anticipate 
a similar relationship for continuation. 



PREDICTING EXPECTED CONTINUATION 

Table 20 presents the results of a multivariate statistical analysis designed to 
test many of the hypotheses proposed in the preceding section.*^ Expected continua- 
tion as reported by the superintendent is the dependent variable regressed on the 
independent variables previously defined with the coefficients shown in standard- 
ized form in Table 20. In broad terms, the results indicate that the district's decision 
to continue a project was based on the superintendent's weighing of four general 
concerns— the project s past success, the centrality or importance Of the educational 
need served by the project, the resources required by the project, and the organiza- 
tional-political forces inhibiting or promoting the innovation. 

In particular, Table 20 shows that the following detailed hypotheses proved 
statistically significant: 

1 The higher the perceived success, the more likely the project was to be 
continued. 

These vanabief ^re listed in Table 19 under the respective names "Initiate projects if additional 
resources" and "Initiat/ projects ifcan be continued " The bivariate correlations of 14 and .29 support 
the above hypotheses 

The advantage of multivariate regression analysis is that the effect of each explanatory factor on 
the dependent variable is computed statistically for all the other factors in the e<juation However, if the 
explanatory factc-s are highly interrelated, the procedure loses some of iU? accuracy Therefore, we could 
not test a number of the hypotheses discussed in the preceding section. 

The statistical procedure whose results are shown by Table 20 is two-stage least squares The firat 
stage was used to estimate the superintendent's perceived success for a project, as discussed earlier 
and the T-statistics for this procedure have somewhat diflferent meanings than in ordinary least-squares 
regression. See R Dhrymes. Econometrics. H-.rper & Row. New York. 1970 Because the e^^timates from 
this two-stage procedure are sensitive to oui assumption about how the superintendent's perception of 
success IS calculated. Table 21 presents regression results that compare the effects of the factors with and 
without estimated project success included. The calculations excluding estimate<i success use ordinary 
least squares ir\^\ 
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Table 20 

FACiORS AFFLC'lINC CONTINUATif^N : ALL PROJECIS AND TITLE III PROJI-XTS 





Standard ized Re^',ros- 
sion Coef f ic lent, 
for Effects on Ex- 
pected Continuation 


Explanatory Factors 


(1) 
All 

Projects 


(2) 
Titl'e III 
Projects 


Project Considerations 

?eT/ceived success (estimated) 
Centrality ^ 
(Funding level 
Resources [Funding per student 

[Project covers high and elementary schools 


-•'8„ 
-.21 - 


. 18 


Institutional ^Characteristics 

/ School district enrollment (log) 
I Percentage of population black 
Setting ^ Percentage^ of population Spanish' 

) Extent of political-social difficulties 
V Difficulty from student test scores 
(Initiate projects if additional resources 
Decision- ) Initiate projects if can be continued 
- making iProject supported by principals 
^ Superintendent 's tenure 


** 

-.13 
.10 

.2} 

-■07,** 
■"„ 
■15„ 
. 1 A 


.2A 
,21 

•"2, 

.20 

■22, 
. 20 


Educational Methods ^ 
Behavioral modification 
Enrichment 

Classroom organizational change 
Intensive traditional staffing 
Administrative change 


- 02 
.02 
.11 
-.07 
-.02 


,01 
,0A, 
. 13 
-.12 
.01 


Federal Programs 
* Right-To-Read 

Bilingual, Title VII 
Vocational Kducat Ion 


-.03 
-.13 




2 

R (based on 2-stagc least squarei;) 




.A5 


Cons tan L 


-3.02 


•kk 

-2,77 


Number of observations 


122 
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\'()H : An asterisk (*) indicates s i g!\ i f icance r.r the ,15 level; a dou^^le 
asterisk (**) induates significance at t!\e .10 level; and a triple asterisk 
(**>V) uviudtes significance at the .01 level. 



^ACIORS AUbCilN«.> tOMlNUAIlON: MOD^'.LS I, II. A.\D III 



Explanatory Factors 



Projecc Considerations 

Perceived succebb ^estimated) , 
Centrality 
I Funding 
Resources Funding ^er student 

jproject covers high and elementary schools 



Standardized Regression Coeflicients 



Model I: 
Without Erfects of 
sSuccess , Educational 
Methods, Program 



.27 



Model II: 
Without Effects of 
Educational Methods 
and Programs 



.36 
.21* 

a;* 



Ail 
Factors 



. 

.23'' 
.18* 
.18" 
.21* 



Institutional Chai^cteristics 

/ / School district enrollment (log) 
I Percentage of population black 
Setting; ' Percentage o'f population Spanish 

I Extent of political-social difficulties 
\ Difficulty from student test scores 
/Initiate projects if additional resources 
I Initiate projects if can be continued 
I Project support'^d by principals 
Superintendent's tenure 



Decision- 
making 



.27 

.13 

.22 
-.08 

.15 



.22 

.10 

.28 
.06 
27* 



.25 * 

.13* 

.10 

.08 

.23^ 

.07 

.25* 

.15^ 

.lA*" 



bducational Methods 

Behavioral modification 
Enrichment 

Classroom organizational change 
Intensive traditional staffing 
Administrative change 



Federal Programs 

Right-To-Read 

Hi Ungual. TitK- VI 1 

Vot at lr>t>al Pdur^ition 



R 

Number ot obsurvat i->n.s 



122 



.AO 



.02 
.02 
.11 
.07 
.02 



-.1 I 



NO'.}-: An asterisk (*) ind*eates signiu^anLe at i A'y levt-l; «i double asterisk , .md i cau-s 
signiti.an.,' . the . K> lovei •, and a triple asterisk indicatt-s s i >;n i f icance at the .01 level. 
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2 Projects that were central to the educational goals of the district were 
more likely to be continued. 

3. The more expensive the project, the less likely was its continuation. (This 
finding will receive further analysis during the second phase of this study; 

^the results may be sensitive to whether projects having large capital expen- 
ditures ?.bsorb that capital (e.g., in the form of a resource center or a 
computer) and thus do not need to be "contirued.") 

4. The larger the district, the more likely ras project continuation. (Appen- 
dix C suggests possible explanations for this finding.) 

5. The higher the percentage of blacks in the district population, the less 
likely was continuation. (This finding was not significant for our sample of 
Title III projects.) 

6. The higher the percentage of Spanish-speaking people in the district, the 
more likely was continuation. This positive affect of percentage of vSpanish 
on continuation reflected the likely continuation of Bilingual projects 
whetliciunded by Title VII or Title Iff 



7. General^litical-social difficulties were not significant (perhaps due to 
poor measurement), but difficulty from student test scores— a sensitive 
indicator of community pressure— was positively related to continuation. 

8. Districts that initiated projects with the intention of continuing were like- 
ly to continue their projects, all other things being equal (e.g., resources 
and success). Such previous intenlion may be indicative of a problem- 
solving approach on the part of the district. 

9. Projects supported by principals were likely to be continued. 

10. The more tenure the superintendent had in the district, the more likely 
projects were to be continued. (The superintendent knows the political- 
bureaucratic "ropes " See App. C for further discussion.) 

In addition to the above hypotheses. Table 20 also presents results for the 
various educational methods and the federal change agent programs. The earlier 
analysis of program comparison? and, variations of the level of use of educational 
methods within the federal programs did not control for other explanatory factors. 
The results afler controlling are generally consistent with the earlier findings. 

Of the various educational methods, the only significant factor is classroom 
organization change. Although the effect is only marginally significant, it suggests 
a commitment within some LEAs to pursue a type of innovation that replaces rather 
than adds on to standard teaching practices. (The negative tendency for intensive 
staffing suggests that teacher aides and assistants paid for by federal funds might 
not be continued on local funds.) 

The federal change agent program comparisons also reinforce the earlier 
findings. Vocational Education, whose projects were judged by superintendents 
among the most successful, had a negative effect on continuation (relative to the 
average Title III projects). Since the analysis of Table 20 controls for such factors 
as estimated project success, project cost, and project importance, the low continua- 
tion likelihood of Vocational Education appears to have been attributable to the 
program itself and the way its projects were perceived by the district. An explana- 
tory hypothesis sugg^'^ted earlier is that many of these projects were viewed as tem- 
porary add-ons and dre not accorded high priority in the district. 
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In contrast, although Title VII projects were seen to be among the least success- 
ful and the most difficult to implement, they tended to have a positive effect on 
continuation."" This finding, considering the various explanatory controls, lends 
support to the interpretation that bilingual education had developed local demand 
for its continuation with or without federal funding. 

Right-To-Read projects had about the same continuation likelihood as the aver- 
age of Title III projects. The data of Table 20 suggest that the continuation of these 
projects primarily depended on explanatory variables other than the program itself. 
Thus, the continuation of Right-To-Read projects may have been particularly contin- 
gent on their perceived success during the trial period and on the political pressures 
in the district for improved student test scores. The effect of explanatory factors on 
the expected continuation of Title III projects is shown in col. 2 of Table 20. The 
results are similar to those analyzed for all projects, except for several expected 
differences. The continuation of Title III projects depended more than the average 
of all projects on the type of educational method (classroom organization changes 
were positive), on the superintendent's tenure (the importance of the superinten- 
dent's support for locally initiated Title III projects was previously hypothesized), on 
funding (more concentrated funding was negative), and on percentage of Spanish- 
speaking population (which reflects the commitment to Title III, Bilingual projects). 



SUMMARY 

The picture that emerges from our analysis of a school district's decision to 
' continue projects is encouraging. Superintendents appear to have been making 
careful choices within their political-economic constraints. If a project was seen as 
central, successful, had the support of the staff", and was not too expensive, it was 
likely to be continued. Such projects were often initiated as the result of the need 
to solve a problem within the district and often replaced' traditional teaching activi- 
ties with new departures (e.g., classroom organizational changes). Projects initiated 
in an opportunistic fashion often became add-ons and disappeared with the termina- 
tion of federal funds. 

»" The shift coefficient for Bilingual would be higher and significant if the control variable "Percent- 
age of population Spanish" were not included in the analysis. Note that percentage of Spanish is signiti- 
cant m the analysis of Title III projects, where it is virtually a surrogate for Title HI, Bilingual projects 



VI. CONCLUSIONS 



As one component of a larger study of change agent projects, this report has 
examined only selected aspects of the highly complex problem of innovation. We 
have, using statistical analyses of survey data, described and identified factors affect- 
ing the implementation outcomes of innovative projects. In particular, we addressed 
three questions: 

• To what extent did the differences between the federal change agent pro- 
grams affect implementation outcomes and continuation? 

• Which characteristics of innovative projects significantly affected im- 
plementation and continuation? 

• Were differences in institutional settings related to variations in im- 
plementation and continuation, and, if so, which institutional character- 
istics had significant effects? 

This section reviews the findings in general terms and places them into a somewhat 
broader perspective. 

Before discussing the general results, it is appropriate to remind the reader of 
several limitations of the analysis. This was an exploratory study; there were meth- 
odological problems, and caution needs to be exercised in interpreting the results. 
Accordingly, this section reviews findings about which we feel the most confident. 
One particular methodological decision should be recalled — namely, the compari- 
sons of federal programs dealt with the implementation and continuation of projects 
sponsored by the programs. The federal change agent programs have goals other 
than the ones assessed here. Finally, this volume complements the case study 
materials reported in Vol. Ill, and thus the two volumes should be read in concert. 
Volume IV synthesizes the findings of these quantitative and qualitative approaches 
and, moreover, offers policy conclusions that are beyond the purview of this report. 



FEDERAL PROGRAM EFFECTS 

The management strategies and substantive priorities of the federal change 
agent programs differ in important ways. These strategies range from the non- 
categorical, locally initiated Title III program to *the categorical and somewhat 
prescriptive Right-To-Read; from the state administration of Title III St-ate Plan and 
the state portion of Vocational Education, Part D, to the federal administration of 
Title III, Section 306, Right-To-Read, Title VII (Bilingual), and the federal portion 
of Vocational Education. However distinctive these strategies seem to be, the differ- 
ences between the federal programs accounted for little of the variations in projects. 

The federal programs exercised influence primarily on the initial design of 
innovations. Each of the categorical programs not only succeeded in getting districts 
to try projects in their substantive areas but also might have induced higher levels 
of intensive staffing, enrichment activities, behavioral modification techniques, and 
school wide administrative changes than would otherwise have been attempted with- 
out federal funds. The locally initiated, noncategorical, and competitive Title III 
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programs have resulted in a wide diversity of innovations, some of which are ambi- 
tious cha%e efforts, whereas others appear peripheral to the district's perceived 
priorities. 

Another differential but unintended effect of the federal programs involved 
difficulties in implementing innovations. Each program appeared to generate prob- 
lems that reflected its particular selection mechanisms, guidelines, substantive pri- 
orities, and expectations. Thus, Vocational Education projects seemed to experience 
difficulties gaining support within the district and within the community, whereas 
Bilingual projects gained support but were the most difficult to implement both 
because they lacked bilingual materials and staff and tended to have unrealistic 
goals and timetables for change. Right-To-Read*s prescribed materials and methods 
appeared to result in adaptation requirements that were not anticipated by these 
prescribed plans. In contrast, State Title III projects experienced difficulties associat- 
ed with ambiguous and overly ambitious goals. 

These contrasting strains of limited leverage on project design accompanied by 
intrinsic implementation problems^ggest that federal policy may face implicit 
tradeoffs in using the policy instrument of differential funding. Because understand- 
ing the nature of these tradeoffs requires an analysis of the process of implementa- 
tion and its relationship to initiation, we defer further discussion until Vol. IV. 

Whatever intended or unintended consequences the different federal manage- 
ment strategies may have had on project initiation, these differences appeared to 
have little effect on project implementation. The gross comparisons of the average 
project outcomes reveal few important differences between projects sponsored by the 
different programs. This finding suggests two conclusions. First, the differences 
between the programs did not prove significant because their common policy instru- 
ment of differential funding did not have a major influence on implementation. 
Second, factors other than federal policy account for most of the variance in project 
outcomes, and these factors were only marginally influenced by current federal 
policy. 

Volume IV suggests some policy implications from the above conclusions. Next, 
we will consider factors other than federal policy that affected implementation. 



PROJECT CHARACTERISTICS AFFECTING 
IMPLEMENTATION 

Many project evaluations focus on educational treatments and attempt to relate 
them to student outcomes. These efforts may be misguided. Educational treatments, 
defined solely in terms of their technology or method, were only weakly related to 
implementation outcomes because other elements of project design had stronger 
effects. The analysis showed that a project's implementation strategy significantly 
affected implementation. This finding is subject to measurement errors and the 
difficulty of reliably measuring diverse educational techniques. Nonetheless, we can 
safely conclude that federal policy, as well as local designers and evaluators, should 
concern itself with fostering appropriate implementation strategies. 

In addition to establishing the importance of implementation strategies, the 
quantitative analysis suggests characteristics of an implementation strategy (i.e., a 
particular mix of strategic choices) that might be more likely to result in high levels 
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of teacher change and achievement of project goals. In particular, frequent staff 
meetings, staff training held in conjunction with meetings, and project requirements 
placed on teachers to change their practices seemed to be elements that worked 
together so that project participants could adapt to the project and vice versa. We 
hypothesize that this implementation strategy, which might be called a mutual 
adaptation strategy, may be characteristic of innovations that result in significant 
change. 

Volume III provides a more detailed understanding of how and why a mutual 
adaptation strategy works in practice. However, the quantitative evidence docs 
suggest that the significance of a mutual adaptation strategy may be in the project's 
interaction with its institutional setting, as originally hypothesized in Vol. I. 



EFFECTS OF INSTITUTIONAL SETTING 

The analysis clearly establishes the importance of the institutional setting for 
the outcomes of an innovative project. Although only some local characteristics were 
measured— and these in gross terms— the setting explained more variation in 
project outcomes than any other factor. 

The major significant variables associated with the institutional setting reflect- 
ed organizational climate. We found that a supportive setting facilitated teacher 
change and the achievement of project goals. In light of this result, the importance 
of a mutual adaptation implementation strategy becomes apparent. It aids the local 
school organization's adaptation to the project and vice versa. Generally speaking, 
the conclusion, which receives further attention in Vol. Ill, is that the effective 
implementation of innovative projects depended primarily on a supportive institu- 
tional setting and on an implementation strategy that fostered the mutual adapta- 
tion of the staff to the project's demands and of the project's design to the reality 
of its setting. 

The dominant importance of the institutional setting for project outcomes sug- 
gests an explanation of why the federal policy instruments employed by the change 
agent programs may have had so little effect. To affect outcomes in major ways, 
federal policy would have had to influence the motivations of local innovators more 
strongly than it has. Perhaps federal policy could develop such leverage by providing 
incentives that promote a supportive institutional setting or by awarding grants 
only to those LEAs that make the necessary commitment. These issues will be 
considered in Vol. IV. 



CONTINUATION HYPOTHESES 

An important measure of the effectiveness of federal change agent policy in- 
volves continuation— the extent tc which project activities are continued by the LEA 
in whole or in part afler federal funding ends. Our analysis of continuation was 
limited for several reasons. First, we studied projects in their last years of federal 
funding and thus could only examine expected continuation. The next phase of 
Rand's research will focus on projects afler the withdrawal of federal support. In 
addition, the quantitative analysis of continuation was concerned with the district 
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level rather than with the classroom level. Volume III treats continuation from the 
standpoint of the extent to which individual teachers incorporated project activities. 
Finally, the measurement problems inherent in this exploratory study limited the 
analysis of continuation. In short, we view the findings as hypotheses to be tested 
in the next phase of research. 

Despite the above reservations, the data provide strong evidence that superin~ 
tendents weigh four general concerns in reaching a decision about continuation: 
the project's "success" during implementation, the centrality or importance of the 
educational needs served by the project, the resources required by the project, and 
the organizational-political forces inhibiting or promoting the innovation. Federal 
policy leverage in any of these areas seems limited but may be particularly ineffectu- 
al in influencing local organizationalrpolitical concerns. The challenge for designers 
of federal policy is thus clear even though the solutions are not. These themes will 
be examined in Vol. IV and in the final reports of Rand's research on the educational 
change agent programs. 
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Appendir A 



SURVEY DESCRIPTIONS, SURVEY SAMPLING DESIGN, 
AND SAMPLE REPRESENTATIVENESS 



This appendix outlines the instruments used for the nationwide survey of 
change agent projects, describes the survey sample design, and examines its repre- 
sentativeness compared with the population of all school districts. 



SURVEY DESCRIPTIONS 

Thirteen data-gathering instruments were administered in the survey: 

Core Questionnaires 

1. Superintendent (60 minutes) 

2. Federal program manager (60 minutes) . 

3. Project director (90 minutes) 

4. Principal (60 minutes) 

5. Teacher (40 minutes) 

Supplements 

6. Title III (administered to project director only; 15 minutes) 

7. Bilingual (administered to project director, principal, 
teacher; 15 minutes) 

8. Right-To-Read (administered to project director, principal; 

15 minutes) 

9. •Vocational Education— project director (15 minutes) 

10. Vocational Education— principal (15 minutes) 

11. Vocational Education— teacher (15 minutes) 

Information Sheets (self-administered) 

12. District information sheet (filled out by superintendent's 
office) 

13. Project information sheet (filled out by project director) 
A list of the topics covered in each instrument follows: 

1. Superintendent Questionnaire 

A. General questions 

District financial situation 
School board role 

Administrative structure of district 
Educational climate (including recent controversies) 
Use of federal funds in district 
Attitude toward innovation 
Non-specially funded innovations 
Federal versus state funding 
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B. Project-related questions 

(If district had two projects in sample, this set of questions was asked 
about both projects. In the few largest districts with more than two 
sample projects, Rand selected two projects to ask the superintendent 
about.) 

Origins of project 
Effects on district image 
Centrality to district goals 
Community reactions 
Role of project director 
Perceived success of project 
Dissemination 
Continuation 

C Personal background 

D. Checklist of previous innovations in district 

2. Federal Program Manager Questionnaire 

A. General questions 

Comparison of four federal programs (Title III, Title VII, 

Right-To-Read, Vocational Education, Part D) 
Szsite versus federal funding 
District use of federal funds 
Non-specially funded innovations 

B. Project-related questions 

(If district had two projects in sample, this set of questions was asked 
about both projects." In the few largest districts with more than two 
sample projects, the federal program manager was asked about the same 
two projects as the superintendent.) 

Origins of project 

Implementation 

Conflicts 

Dissemination 

Continuation 

Perceived success * 

C. Personal background 

3. Project Director Questionnaire 

A. Project-related questions 
Origins of project 
Planning 
Implementation 
Selection of project sites 
Staff training 
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Use of funds 

Goals and techniques of project 
Conflicts 

Technical assistance 
Community involvement 
Support of district administration 
Dissemination 

Perceived success/ reasons for 

Relations yvith federal/state project officer 

Evaluation 

Continuation 

B. Personal background 

4. Principal Questionnaire 

A. General questions 

School characteristics 
School-community relations 
Attitude toward innovation 

B. Project-related questions 

Origins of^project at school 
Teacher selection and training 
Goals and techniques of project 
Principars role in project 
Conflicts 

Community involvement in project 
^ Dissemination (within and outside school) 
Perceived success/ reasons for 
Effects on school 
Continuation 

5. Teacher Questionnaire 

A. General questions 

Attitude toward innovation 
School morale and principal role 

B. Project-related questions 

History of personal participation in project 
Training 

Role of teachers in various aspects of project 

Implementation in the classroom 

Perceived success of project and effects on students 

Continuation 

C. Personal background 

6. Title III Supplement 

Project classification (for field^^r]^ screening) 
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Use of community resources 

IVD (Identification, Validation, Dissemination) instrument 
Fourth-year continuation /dissemination 

7. Bilingual Supplement 

Linguistic background of students 
Project goals 
Teacher training 
Classroom language use 

8. Right-To-Read Supplement 

Previous reading programs 

Effectiveness of various elements of Right-To-Read strategy 
Effectiveness of Right-To-Read planning procedure 
Reading curriculum components 

9. Vocational Education Supplement: Project Director 

Concept of career education 

Project techniques 

Project funding 

Role of funding agency 

Evaluation 

Dissemination 

Effects of project 

10. Vocational Education Supplement: Principal 

Concept of career education 
Teachers' attitudes 
Project techniques and resources 
Effects of project 

11. Vocational Education Supplement: Teacher 

Concept of career education 
Project techniques 
Effects of projects 

Attitudes toward project director and principal 

12. District Information Sheet 

Size of student population and number of schools 
District annual expenditures by category 
Sources of revenue (local, state, federal) 
Teacher salaries 

Current federal projects in district 

13. Project Information Sheet 

Names and characteristics of schools involved in project 
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Number and types of students served 
Project staffing 
Sources of project funding 
Project expenditures by category 

Respondents Who Serve More Than One Role 

In some cases, respondents may combine two or more roles— for example, super- 
intendent and federal program manager, or project director and principal. In that 
case, the following rules applied for questionnaire administration: 

1. If the superintendent was also 

• Federal program manage^ NORC* administered a combination super- 
intendent/federal program manager questionnaire. 

• Project director, Rand decided whether NORC should administer a 
project director or superintendent questionnaire. 

• Principal NORC administered a principal questionnaire. 

• Teacher, NORC administered a superintendent questionnaire. 

2. If the federal program manager was also 

• Project directory NORC administered a project director questionnaire. 

• Principal NORC administered a principal questionnaire. 

• Teacher.. NORC administered a federal program manager question- 
naire. 

3. If the project director was also 

• Principcd, NORC administered a combination principal/ project direc- 
tor questionnaire. 

• Teacher NORC administered a project director questionnaire. 

4. If the principal was also a teacher, NORC administered the principal ques- 
tionnaire. 

5. If the superintendent was also both federal program manager and project 
director or federal program manager and principal NORC in eacli case ad- 
ministered a combination superintendent/federal program manager ques- 
tionnaire. 



SURVEY SAMPLE DESIGN 

Projects of six types from four programs are represented in the change agent 

sample as shown in Table A-1. 

The sampling was accomplished in two stages: a sample of 18 states from the 

contiguous 48 states was followed by the selection of projects within states. The 
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general objective of the state sample was to obtain a sample that was broadly repre- 
sentative of region, level of educational fundingy and intensity of educational man- 
agement at the state level. Subject to these criteria, the selection of states was 
weighted according to population of persons (and hence roughly according to num- 
ber of Title III projects) so that the state sample could provide an adequate pool from 
which to select projects. In addition to the 224 projects, project directors from other 
projects in the preselected districts were interviev^red whenever the opportunity 
arose (increasing the sample of projects by 69). 



Table A-1 
CH/VN'GE AGBNT PROJI-CTS: SAMPLE SIZE 



Sample Size 



Programs ^ 


Prese- 
lected 


0«--Site 
Selection 


Total 


1. 


KSI:A^ Tide III, state 










administered 


132 




176 




BSBA Title III, federally 










administered 


18 




18 


2. 


KSKA Title VII 


25 


16 


41 


3. 


VEA^ Part%D, state 










administered 




2 


16 




VEA Part D, federally 










administered 


10 




10 




Right^-To-Read 


25 


7 


32 


Total 


22A 


69 


-293 



Elementary and Secondary Education Act. 



Vocational Education Amendments. 



Title III projects were selected from among all 18 states. For the other pro- 
grams, subsets of about ten states were chosen; this permitted selection of at least 
two projects per state and therefore some chance of identifying between-state differ- 
ences in these smaller samples. In selecting projectSy the guiding objectV)e was to 
obtain samples that represented the diversity of projects across a number of criteria 
of analytic interest In other words, the selection was to provide projects in school 
districts that were large, small, urban, rural, of varying concentrations of differenf 
racial-ethnic groups, and of varying socioeconomic status. 

The two stages of sampling differ sharply with respect to "sampling philoso- 
phy." The first stage is probabilistic, except for a modification that will be described 
in the next section. The second stage Treely employs judgment methods when it is 
necessary to represent diverse attributes with small samples. 

In the remainder of this section, the population of eligible projects is specified; 
the sampling procedures for states and for projects are explained; and other aspects 
cf the sample design are discussed. 
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The Population of Eligible Projects 

At the time of Rand's pre-sampling data collection activities, the Office of Edu- 
cation files contained project ^ibstracts for nearly all State Title III projects as of J uly 
1972. From these files, coders completed a summary form for each projea that 
included information on its geographical location, size, funding, start date, orienta- 
tion, and target group 

Not all Title III projects fell within the charter of this study. Projects that met 
any of the following criteria were excluded: 

• All first-year projects. 

• All projects with yearly funding of less than $10,000. 

• All projects that served only pre-school or post-high school populations. 

• All projects that were jolely resource centers. 

• All projects that only involved curriculum development. 

• All projects that solely relied on home visits. 

Applying these criteria reduced the population of projects in the 48 states from 1730 
to 1085. These 1085 projects thus comprised the sampling population of eligible Title 
III projects for the study. 

In several respects, the sampling population is not the same as the target popu- 
lation. For example, some small fraction of project directors probably failed to 
submit abstracts to OE. Many projects that began in the fall of 1972 were omitted 
because funding began after the July deadline. On the other hand, some projects in 
the files had since terminated; they were dealt with by replacement sampling. The 
problem of projects inadvertently omitted from the sampling population remains, 
but is significant only to the extent that their experience may systematically differ 
from that of the included projects. 

The project populations for the other three programs were obtained from lists 
provided by program directors within OE. Projects in the other, programs had to 
meet the same criteria for eligibility as were specified for Ti Je III. In some cases, 
there were additional criteria. 

The Sample of States 

Measure of Size. The measure of size (MOS), or weighting criteria, for the 
selection of states was total state population according to the 1970 Census. On this 
basis, six states (California, Illinois, New York, Ohio, Pennsylvania, and Texas) fell 
into the sample with certainty. The MOSs for California and New York were actual- 
ly almost twice certainty. For this reason, state selection probabilities were calculat- 
ed under the temporary assumption that the sample size would be 20: 

o, u A (20)(population, state A) 

Selection nrobability, state A = \ — . . ^ ' 

(population, 48 states) 

Before allocating states into strata and constructing the sampling frame, probabili- 
ties for the two large states were diminished by one (1.0). This adjustment assured 
that 18 discrete states would be chosen (e.g., that California would not be selected 
twice). 
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"^'Stratification. The states were stratified by three separate factors to obtain 
balanced representation by region, level of educational funding, and intensity of 
educational management at ^ the state level. In each case, the stratification was 
defined so that the sum of the adjusted MOSs of states in all strata was roughly 
equalized. In other words, strata containing California and New York each repre- 
sented approximately 30 million persons; all other strata ren^rsented approximate- 
ly 20 million persons. 

The nine regional strata are depicted in Fig. A4 ' -t.^. is labeled with its 
respective MOS. 

To obtain stratification by level of educational /undt«g (elementary and second- 
ary), states were ranked according to cxf)enditure per pupil (dollars per average 
daily membership collected by the National Education Association for the 1971-72 
school year). This list was then divided into nine strala, equalizing as closely as 
possible on MOS. 

As a surrogate variable for intensity of educational management at the state 
level, the following ratio was computed for each state: 

Total expenditures for SEA administration 
Total school expenditures for state 

(Data for the numerator were collected by OE for 1969 and 1970. Denominator data 
were collected by the Advisory Comnilosion on Intergovernmental Relations for 
1969.) States were then ranked and divided into strata in the same manner as for 
the educational funding stratification. 

Sample Selection. A three-dimensional sampling frame was employed, with 
each dimension corresponding to a stratification factor. For each stratification fac- 
tor, ther0 were nine strata, or levels. The result can be characterized as a 9 x 9 x 
9 matrix with 729 cells. States were allocated among the cells of the matrix accord- 
ing to their respective strata locations. The result was 46 occupied cells and 683 
unoccuoied cells (two cells accommodated two states each). The problem was 
select two occupied cells from each row, each column, and each file of the matrix, 
and to do so with probabilities equal to the combined state selection probabilities in 
each cell.^ The 18 states thus selected are listed below: 

* The algorithm for accomphshing this was* presented by Jessen [1] under the label of Probability 
Lattice Sampling, Method 2. 

Briefly, the task involves arbitrarily designating a "feasible set*' of possible samples, each sample 
satisfying the marginal constraints (i.e , two cells from each row, column, and file). Selection probaDilities 
are designed to each successively designated sample in accordance with certain decision rules. When the 
sum of selection probabilities thus assigned totals 1.0, the feasible set is complete and no more sam^ j 
should be designated. Thevdecision rules guarantee that the sum of sample selection probabilities for all 
samples containing any particular cell is «qual to the respective cell probability. After the feasible set 
IS complete, one of the samples is selected probabilistically, observing the assigned probabilities If any 
cell in the selected sample contains more than one state, one state is selected with probabilities prooor- 
tional to state MOS. The result is a sample that observes the marginal constraints, but in which states 
have been selected with probabilities proportional t^ state population. 

This procedure was modified somewhat in its actual implementation. Empirical research has sug- 
gested that in the set designation process, the sets first designated tend to contain fewer aberrant coHb 
than samples subsequently designated [2], It is also more difficult to maintain the marginal constraints 
in the later samples. In other words, samples tend to be less representative as set designation proems. 
To take advantage of this procedural artifact in the change agent sample, feasible set designation was 
terminated after the sum of sample selection probabilities totaled .5. 
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Arkansas, California, Colorado, Florida, Georgia, Illinois, Kentucky, iMassa- 
chusetts, Michigan, Missouri, New Jersey, New York, North Ciirolina, Oiiio, 
Pennsylvania, Texas, Wasiiington, Wisconsin. 

Selection of Projects within States 

In general, a factorial design might proceed as follows. For each program, fac- 
tors (i.e., variables) that influence project effectiveness and for which data are readi- 
ly available are identified. Two or more levels are designated for each factor. By 
combining factors and their respective levels, a matrix is formed that has as many 
dimensions as there are factors. Projects are selected from each cell of the matrix. 
Thus, the resulting sample may not be probabilistically representative of the total 
population of projects, but it will represent the diversity within that population with 
respect to the factors that have been identified. Although the factorial arrangement 
of the sample observations is more difficult to achieve in a survey situation than in 
the usual experimental one, it can be done approximately;^ when done it will provide 
for more efficient analysis of not only main effects of factors but also of various factor 
combinations, or interactions. In view of the smallness of the samples employed in 
the change agent study, the factorial approach seems particularly well suited. 

The procedure described in the foregoing paragraph was formally implemented 
for the Title III projects where the population size was large and the sample size 
relatively large. For other categories, where both samples and project populations 
were small, c^election was more informal and somewhat judgmental, but the overrid- 
ing criterion was still to obtain samples that represented the diversity within ihe 
respective populations. , - 

The factors dealt with explicitly in the sample design are those for which data 
were readily available. These factors tend to be of demographic and economic na- 
ture, and therefore perhaps not directly manipulabie by federal policy. The basis for 
selecting the design factors, however, was to ensure representation for different 
levels of the more policy-relevant factors. Conventional procedures of analysis can 
be used on the sample data to study the effects of the other factors. In essence, the 
effects of the design factors will have been "balanced out" and should therefore 
improve the effectiveness of analysis. 

Title III, State Administered. One hundred thirty-two state^administered 
Title III projects were selected in six samples. The first was a sample of nine "man- 
agement" oriented projects drawn from a sampling frame encompassing all 18 
states. The second was a sample of 63 ^'classroom" oriented projects from another 
frame encompassing all 18 states. These two samples combined provided four pro- 
jects from each state; data from these samples will provide a basis for generalizing 
behavior across states, and for identifying general between-state differences. There 
were additional samples from each of four large states, with 15 projects each. The 
additional intensive sampling was required in order to enable more detailed analy- 
ses of behavior for which between-state differences cannot be accommodated by 

' Since some cells in the universe of projects may not be occupied by projects, for whatever reaiX)n, 
it is not possible to achieve a perfectly balanced sample. This causes complications both in setting up the 
design and in the analysis. However, they can be dealt with by various techniques Adjusting boundaries 
on levels where factors are continuous to obtain a better "filling of cells" is one measure. Use of "missing 
data'* techniques in the analysis is another. 
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simply adding an additional variable to the functional form. In such cases, it may 
be difficult to pool data across states, so it is desirable to have a few states with 
sufficient sample sizes to support state-specific analyses. 

Thus, six sampling frames were required: two frames encompassing all 18 
states, which overlapped four state-specific frames. The six frames were each con- 
structed according to the factorial approach noted above. A data file was assembled 
for each of the projects in the sampling population, drawing from Bureau of the 
Census data aggregated at school district level. The variables in the file were exam- 
ined to determine which might be used as control factors in the sample selection. 
The primary basis for the determination was the usefulness of the variable in 
partitioning the projects into groups that are of interest from the standpoint of 
analysis Correlation matrices of the potential factors were also examined to ensure 
that chosen factors would not be overly redundant. The chosen factors are listed in 
Tablo A-2 along with the number of levels for each factor for each of the six samples. 



Table A-2 

NUMBKR OK M-VEJS FOR STATE TITLE ill SAMPMNG FACTORS 





18-State 


Sample 


Intensive 


Sampling 




Manage- 


Class- 


Calif- 


Illi- 


New 


New 


I tern 


ment 


room 


ornia 


nois 


Jersey 


York 


District enrollment 


3 




3 


3 




2 


Urban/rural per 














capita income 


3 


3 


3 


3 


3 




Race (perc'entage black) 








15 


15 . 


15 


Sample size 


9 


63 


15 



Of^ the 674 identified eligible projects in the 18 states, 80 were designated as 
being management oriented. These 80 were distributed among the cells of a two-way 
matrix, using the Enrollment-Urbanity and Income factors; nine projects were sam- 
pled from this frame. 

The remaining 594 classroom projects were distributed among the cells of a 3 
/ 4 X 4 matrix using all three factors. From the 41 occupied cells of this matrix, 
63 projects were sampled in the following manner. First, subject to the quota of four 
projects per state (including management projects), two projects were selected at 
random from each cell containing over 20 projects, and one project was selected from 
all other cells; this resulted in 47 selections. Next, state quotas were filled by random 
draws, beginning with the most sparse states; insofar as possible, care was taken not 
to exceed two selections for any cell, and to make only one selection for any cell from 
a given state. The result was a sample of 72 projects, with four projects from each 
sampled state; 63 of the 72 were classroom identified projects distributed fairly 
evenly over a 3 > 4 / 4 factoi ial with seven empty cells; nine of the 72 were 
management identified projects distributed over a 3 x 3 factorial with one empty 
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The more intensive sampling in four states (California, Illinois, New Jersey, and 
New York) proceeded as follows. For each state, classroom projects were allocated 
among the cells of a two-dimensional matrix, using: the Enrollment-Urbanity and 
the Income factors. After noting where the four projects already selected fell in the 
matrix, 15 additional projects were sampled so that the total 19 were evenly distrib- 
uted over the matrix. As with the 18-state sample, the sample size for each cell was 
somewhat arbitrary, although roughly proportional to cell size; selection within 
cells was random. Note in Table A-2 that the frame for New York is more properly 
characterized as simple stratification rather than factorial; this is because of the 
dominance of New York City. It should be observed that the projects in the New 
York City stratum were further stratified by borough. 

The sampling population of 674 projects was not the same as the target popula- 
tion, and for this reason many projects in the original sample of 132 were found to 
be ineligible on closer inspection. Twenty dealt with mentally or physically hand- 
icapped children, and another ten failed the eligibility criteria for other reasons. On 
verifying our selections with State Title III coordinators, we learned that an addi- 
tional 30 projects had been terminated. Replacements were selected for the ineligi- 
ble sample projects in such a manner as to maintain the state quota and, insofar as 
possible, to maintain the distribution across the ceils of the sany)ling frame. This 
effort was hampered because the need to rush information to the fieldwork subcon- 
tractor necessitated making replacements in several iterations rather than in a 
single operation. For some states, the replacement pool was soon depleted, but we 
were able to obtain from State Title III coordinators lists of existing second-year 
projects that had been excluded from the 1972 annual report to USOE, the source 
of the original sampling list. 

Title III, Fcierally Administered. Eighteen projects were selected from 
among those federally administered Title III projects with yearly funding exceeding 
$50,000. This funding floor effectively limits selection to categories of projects that 
have been identified by USOE Title III officers as being of prime concern, at least 
from the standpoint of time consumption. Eliminated are SWRL^ curriculum im- 
plementation projects and reading readiness projects. 

Nine states were selected from the 18 sample states, using regional stratifica- 
tion: California, Florida, Illinois, Missouri, New Jersey, New York, Pennsylvania, 
Texas, and Washington. 

The federally administered Title III projects in these states were allocated 
among the cells of a 3 X 3 matrix, using the Enrollment-Urbanity and Income 
factors. A sample of 18 projects was sampled such that each cell was represented at 
least once, and each of the nine states was represented twice. Four replacement 
projects were subsequently required, but this was accomplished without altering the 
sample structure. 

Title VII, Bilingual. Of the 18 sampled states, only eight contained more than 
two Title VII projects; the Title VII sampling was restricted to these eight: Califor- 
nia, Colorado, Massachusetts, Michigan, New Jersey, New York, and Texas. 

The allocation of sample projects among the eight states was roughly propor- 
tional to the sampling population distribution. The sample was also allocated among 
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target language groups roughly proportional to their incidence in the sampling 
population: 

Mexican-American . . 
Other Spanish language 

Indian 

French/Portuguese . . 

Chinese 

Total ^ 30 

The number of distinct projects was 25, since some were trilingual. For the Spanish 
language projects, a spread was obtained over different degrees of urbanization. For 
the other projects, the sampling populations were too sparse to permit further 
sample control. 

VEA, PartD: Vocational Education. It was intended to select 25 VEA, Part 
D projects in ten states. The following sample characteristics were desired: ten 
federally administered projects, representing districts that are large and small, 
urban and rural, and some in Model Neighborhoods; 15 state-administered projects, 
with at least one, from each state, and all concentrating on the Career Education 
component (th^Tederally administered projects all have this concentration); no 
projects tha^operate exclusively in high school years. Given these sampling con- 
straints, the selection of states was more a process of elimination. Even so, only 14 
distinct districts were obtainable for the state-administered projects. The selected 
states were Arkansas, California, Georgia, Kentucky, Missouri, New Jersey, New 
York, North Carolina, Ohio, and Washington. 

Right-To-Read. Project sites for the Right-To-Reaci program are designated as 
either Impact Site, Transition Site, or Expansion Site. The so-called Impact projects 
include at least one of each site type, and other projects generally have only one site. 
Only 12 of the 18 sampled states containe^ Impact projects. Two of these 12 were 
eliminated at random to provide the ten'states for the Right-To-Read sample: 
California, Georgia, Illinois, Massachusetts, Missouri, New Jersey, New York, Ohio, 
Wisconsin, and Texas. 

One Impact project and one single-site project were selected from each of ten 
states. Additional single-site projects were selected from California and Texas to, 
bring the total to 25 projects. 

Sample Summary 

Distribution of sampled projects for all programs across all states is shown in 
Table A-3. 

Sampling Schools and Teachers 

For each project selected for the study we sampled two schools (if more than two 
existed) and two project teachers within each school. 

When the project director was first notified of the impending survey, he was 
asked to list the schools served by the project, ranked according to their entry into 
the program. When more than two schools participated in the project, a random 
sample of two schools was drawn from the list. 

11^ 



12 
8 
3 
3 
4 
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Table A-3 

DISTRIBUTION OF PRESELECTED SAMPLE PROJECTS BY STATE' 





Title III 


Right-To- 




Vocational 


Educat ion 










- 








State 


State 


Federal 


Read 


Title VII 


• State 


Federal 


Total 


Arkansas 


4 


— 


— 


— 


1 


1 


6 


California 


19 


2 


4 


6 


2 


1 


34 


Colorado 


4 




— 


2 


— 


— — 


6 


Florida 




2 


— 


2 


— 




8 
8 


Georgia 


4 




2 


— 


1 


1 


Illinois 


19 


2 


9 








23 


Kentucky 


4 








2 


1 


7 


Massachuse t ts 


4 




' 2 


3 






9 


Michigan 


4 






2 






6 


Missouri 


4 


2 


2 




1 


1 


10 


New Jersey 


1^ 


2 


2 


2 


2 


1 


28 


New York 


19 


2 


2 


4 


1 


1 


29 


North Carolina 


4 








1 


1 


6 


Ohio 


4 




2 




2 


1 


, 9 


Pennsylvania 


4 


2 










6 


Texas 


4 


2 


5 


4 






15 


Washington 


4 


2 






1 


1 


8 


Wisconsin 


4 




2 








6 


Total 


132 


18 


25 


25 


14 


10 


224 



^Table 2 in the text shows the distribution of both the preselected projects 
and the projects selected on-site. 



The selection of project teachers in each sampled school was conducted on-site 
by the NORC interviewers using the so-called Kish Table technique. That is, teach- 
ers were listed in alphabetical order (or some other arbitrary order); then the inter- 
wiewer pre-tabulated random numbers that were keyed to the total number of 
teachers listed. 



REPRESENTATIVENESS OF SURVEY AND FIELDWORIC 
SAMPLES 

This section uses census variables lo characterize the survey and fieldwork 
samples used in the change agent study. To examine the representativeness of the 
samples, it compares the survey sample with the total population of U.S. school 
districts, and compares the fieldwork sample with the survey.* There were two 
reasons for comparing the sample with all districts rather than with all districts that 
have or had funding under the federal demonstration programs: The latter popula- 
tion changes from year to year, and the costs of characterizing it were prohibitive. 

The sampling of change agent projects was accomplished in two stages: A 



* There were 18,655 operating school districts in the United States in the fall 1969 [3]. 
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sample of 18 states from the contiguous 48 was followed by the selection of projects 
within states. The general objective of the state sample was to obtain a sample 
broadly representative on three dimensions: region, level of educational funding, 
and intensity of educational management at the state level. For the selection of 
projects, the guiding objective was ^diversity of school districts— large and small 
districts, urban and rural, varying racial-ethnic concentrations. 

Several criteria were used to determine eligibility of projects for the sample. 
The one most likely to affect the representativeness of the sample may be the 
requirement that the project have a yearly funding level of at least $10,000, which 
would bias the sample to some degree toward larger districts. Because of the weight- 
ing criteria used in selecting the 18 states, six states fell into the sample with 
certainty: California, Illinois, New York, Ohio, Pennsylvania, and Texas. Because 
as a group they contain most of the largest metropolitan areas, we expect our survey 
sample to be biased to some degree toward larger school districts. 

For the fieldwork sample, five fieldwork teams, each %:using on one area of 
innovation, selected projects from 23 school districts for in-depth study in the areas 
of career education, bilingual education, classroom organization, staff development, 
and reading. The districts chosen for each of the five are<^ vary over organizational, 
financial, and political conditions, allowing observation of the effects of such vari- 
ables on the particular innovation. 

The Survey Sample 

Census Characteristics. As our data source we used the 1970 Census School 
District Data Tape which contains the usual census-type variables organized by 
school districts. We used six variables to describe the sample: total population of 
the district, public school enrollment, proportion of the population in the district 
that is urban, proportion that is black, and two income variables— the proportion 
of families with incomes in 1969 of $25,000 or more and the proportion of families 
below the poverty level in 1969. 

From a priori considerations, we used logarithms of both total population and 
school enrollment Transformations for some of the other variables may be sug- 
gested by the statistics for the survey sample. Such transformations may be useful 
in modeling innovativeness of school districts. 

All of the data analysis treated New York City separately. All of New York City 
is one school district, and statistics on the data tape are all aggregated to the whole 
city. However, the city is divided into what are called community school districts, 
which operate with a certain degree of autonomy. Our survey of change agent 
projects included 9 of the 32 community school districts, and data for these 9 com- 
munity districts were not on the census tape. Data for some of the six variables we 
used are available from the New York City Board of Education, and will be discussed 
later. But in the discussion immediately following, we are excluding New York City. 
Our sample, then, consisted of 186 school districts. 

Transformed (to natural logarithms) values are used for population and enroll- 
ment, and untransformed proportions for the other four variables. The means for 
the logs of population and enrollment are 10.580 and 8.990, respectively, correspond- 
ing to population and enrollment values (i.e., exponentiating the mean logs) of about 
39,000 and 800U, respectively. Log enrollment ranges from 5.790 to 13.394, corre- 
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spending to an enrollment range of 327 to about 650,000. The log transformations for 
both population and enrollment produce variables that are very symmetrical and 
nearly normal in distribution, as indicated by the near equality of mean and median, 
and by the measures of skewness and kurtosis. More summary statistics for these 
two variables are given in Table A-4; histograms are in Tables A-5 and A-6. 

The proportion of the population of a school district that is urban covers the 
complete range from 0 to 1 in our survey sample of school districts, with a mean 
proportion urban of .740. The median proportion urban, however, is .931, implying 
some skewness to the left in the distribution. As seen in the histogram in Table A-7, 
just over 40 percent of the districts are between 98 and 100 percent urban. Still, 
about 15 percent of the districts are below 20 percent urban. Additional summary 
statistics for proportion urban are in Table A-4. 

The statistical behavior for proportion black in the 186 survey sample school 
districts is similar to that for proportion urban, on the opposite end of the range. 
Just over 40 percent of the districts have less than 2 percent black. The mean and 
median proportions black are, respectively, .105 and .052, with a range of 0 to .583 
(these are proportions black in the total population of the school districts, not black 
enrollments in the districts or individual schools). Additional summary statistics 
and a histogram are in Tables A-8 and A-9, respectively. 

Finally, we consider the two income variables: proportion of families over 
$25,000 income and proportion of families below the poverty level. Their means are, 
respectively, .046 and .107. They are both somewhat skewed to the right, with a few 
extreme values. Proportion over $25,000 has a maximum of .359, but only 8 percent 
of the districts are over .100. Similarly, the proportion of families in poverty is as 
high as .469, but only 10 percent of the districts are over .200. (See Tables A-8, A-10, 
and A-11.) Both income variables are rather sharply peaked, especially the propor- 
tion over $25,000, which is between .02 and .05 for half of the districts. 

For an indication of relations among these six variables in the sample, see Table 
A-12. 

. The shape of the distribution for proportion of families over $25,0(X) suggests a 
possible transformation for use in the modeling of innovativeness. A square root 
transformation, for example, sharply reduces the kurtosis. It also brings the mean 
and median much closer together. Square root transformations on proportion poor, 
proportion black, and proportion urban have similar effects, although not as dramat- 
ic as with proportion above $25,000. 

We now consider how these sample statistics compare with national averages 
for these variables. 

Representativeness. Although our sample was not chosen to be representa- 
tive of all school districts, we would like to know the distributions of the six variables 
for all school districts in the United States in order to get a sense of how this change 
agent sample deviates from the total district population. 

We used estimates of the national means for the six variables, gathered from 
a variety of sources. They are national aggregate figures, not means over school 
districts, and we must rely on the sample itself for variance^ovariance estimates. 
From Ref 4 we have national means for proportion of population that is urban in 
1970 of .735, ami for proportion black .111. From Ref 5 we have estimates for 
proportion of families over $25,000 income in 1969 of .063, and for proportion poor 



Table A-4 



SURVEY SAMPLE SUMMARY STATISilCS: LOGS OF SCHOOL DISTRICT POPULATION, 
ENROLLMENT, AND PROPORTION OF POPULATION URBAN 



SURVEY SAMPLE OF SCmCOL DISTRICTS 

FILE NONAME (CREATION CATE ' 06/21/74) 



06/21/74 



PAGE 



VARIABLE 
MEAN 
STO OLV 
SKEMNESS 
MAXIMUM 



LOGPOP LJG OF SCHOOL DISTRICT POPULATION 

10.^80 STD ERROR 0*111 

U516 VAf^lANCE 2.297 

C.46D RANGE 7»a74 
15.037 



VALID 08SEPVATICNS - 
MISSING OBSERVATIONS * 



186 
D 



HEDUN 

KURTDSIS* 

HINIMUM 



10.406 
0.027 
7.163 



OK 



0.0 Pcrcent of total 



VARIABLE 
MEAN 
STO DEV 
SkEwNESS 
MAXIMUM 



LCGENRL LOG OF SChOuL DISTRICT ENROLLMENT 

8.990 STD ERROR 0.108 

U477 VARIANCE 2.181 

D.hI? range 7.O04 
1 3.394 



VALID OBSERVATIONS 
MISSING OBSERVATIONS 



186 

0 OR 



MEDIAN 

KURTOSIS^ 

MINIMUM 



8.855 
0.043 

5.790 



0.0 PERCEi^T OF TOTAL 



VARIABLE PCURB PROPORTION OF POPULATION THAT IS URBAN 

^EAN 0.740 STO ERROR 0.025 MEDIAN ^ 0.931 

STD DEV D.340 VARIANCE 0.120 RURTOSIS* 0.057 

SKEHNESS -1.220 ft*NG£ 1.000 MINIMUM O.X) 

MAXIMUM 1.000 

VALID OBSERVATIONS - 186 

MISSING OUSERVATIONS - 0 Oa 0.0 PERCENT OF TOTAL 



^'The measure of kurtosis in all the tables in this report was calculated as 

i l [(X. - x)/s]^ - 3. 
^ i-1 ^ 

Thus, positive values indicate a distribution more peaked than the normal distribution, and 
negative values indicate a distribution flatter ^rmal . 
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Table A-5 



SURVEY SAMPLE HISTOGRAM: LOG OF SCHOOL DISTRICT POPULATION 



SURVEY SAMPLE OF SCHOOL DISTRICTS 6. 06/21/74 PAGE 

FILE NONAME (CREATION GATE » 06/21/74) 
VARIABLE LCGPOP LOG OF SCHOOL DISTRICT POPULATION 

CODE 

I 

1.00 »#♦♦♦»»♦»♦♦» { ii) 5,9 pCT 

I LESS THAN 8.5 



2.00 *♦*♦♦♦♦*♦*♦•*•»♦♦»»»»» I 59)' 31.7 PCT 

I 8.5 TO 9.999 



3.00 ( 26.9 PCT 

I 10 TC 10.999 



I 11 TO 12.499 



5.00 »♦♦♦♦♦♦♦♦♦♦»»♦»♦»»»»»»» ( 22) 11.8 PCT 
I 12.5 OR ABOVE 

I I 1 1 *..I.... i I I I I 

0 10 20 30 40 50 OO 70 80 ^^0 

FREQUENCY 



VALID OBSERVATIONS - 186 
MISSING OBSERVATIONS - 0 



ERLC 



Table A-6 

SURVEY SAMPLE HISTOGRAM: LOG OF SCHOOL DISTRICT ENROLMENT 



SURVfcY SAMPLE OF SCHOOL OISTRICTS' 
Flte NONAMB (CREATION DATE = 06/21/74) ' 
VARIABLE LOGEN^^L LOG OF SCHOOL DISTRICT EMROLLMENT 
CODE 



6. 



06/21/74 



I 

1*00 { 

LESS THAN 7 



ie>) 8.1 PCT 



2.00 



3.00 



4.00 



5.00 



********** ( 

7 TJ a. 499 



59) 31.7 POT 



8.5 TO 9.499 



Pi) 27.4 POT 



^^tf^m^^t^m^*********** *********************** ( 
9.3 TO 10.999 



421 22.6 PCT 



^ «««««« 4t4c4c«4i«4t { 

II CR ABOVE 



19) 10.2 PCT 



0 10 
FRE0U6NCY 

VALID OBSEPVATICNS - 
MISSING OBSERVATIONS - 



20 



186 

0 



30 



40 



50 



..I. 

oO 



• 1. 

70 



PAGE 



90 



ERIC 



1 



Table A-7 

SURVEY SAMPLE HISTOGRAM: . PROPORTION OF POPULATION URBAN 



SURVEY SAMPLE OF SCHOOL DISTRICTS 6 
FILE NONAME (CREATION DATE = 06/21/74) 

VARIABLE PCURB PROP0ATI0>J OF POPULATION THAT IS URBAN 

C00£ 

. I 

27) 14.5 PCT 



06/21/74 



PAGE 10 



1*00 ««««««««««««*^«««4t««««««t««« ( 

LESS THAN .2 ' 



2.00 



3.00 



4i00 



5.00 



.2 TO .599 



.6 TO .899 



19) 10.2 PCT 



38/ 20.4 PCT 



.9 TO .9799 



2c) 14.0 PCT 



«««« «««««« «««««««« ( 

.98 CR ABOVE 



76) 40.9 PCT 



I. 

0 10 
FREQUENCY 



».I. 

20 



>.I . 

30 



..I. 

40 



..I. 

50 



..I. 



« I. 

70 



90 



• .I 

100 



VALID OBSERVATIONS - 
MISSING OBSERVATIONS - 



186 
0 



ERIC 



Table A-8 



SURVEY SAMPLE SUMMARY STATISTICS: PROPORTIONS OF POPULATION BLACK 
WITH INCOME ABOVE $25,000 AND BELOW POVERTY INCOME 



SURVEY SAMPLE OF SCHOOL DISTRICTS 

FILE NONAME {CREATION DATE - 06/21/74) 



6. 



06/21/74 



PAGE 



VARIABLE 
ME AM 
STO OEV 
SKEWNESS 
MAXIMUM 



PCeLK 



PAOPORTIO^ OF POPULATION THAT IS BLACK 



0. IDS 
0. 13^ 
1.3/9 
O.S8 3 



VALID OBSEPVATICNS 
MISSING OBSERVATIONS 



STO ERAOR 

VARIANCE 

RANGc 



186 
0 OR 



O.OLO 
O.OLd 
0.t>B3 



0.0 PERCENT OF TOTAL 



HEDIAN 

KURTOSIS 

MINIMUM 



0.052 
lo952 
0.0 



VARIABLE 
MEAN 
STO OEV 
SKEMNESS 
MAXIMUM 



FATCATS 



Pi^OPO^^TION UF FAMILIES WITH INCOME AdOVE t XiTOOQ 



0.0^6 
0.043 

0.359 



VALID OBSERVATIONS 
MISSING OBSERVATIONS 



186 

0 



STU ERROR 

VARIANCE 

RANGE 



U.U03 
0.002 

Oi,359 



OR 



0.0 PERCENT OF TOTAL 



HEOIAN 

KURTOSIS 

MINIMUM 



0.0B6 
17.438 
0.0 



VARIABLE 
MEAN 

STO OEV 

SKEWNESS 

MAXIMUM 



PC POOR 



PROPORTION OF FAMILIES BSLOil POVERTY INCD^Afc- 



0.107 
0.073 
1.873 
0.469 



VALID OBSERVATICNS 
MISSING OBSERVATIONS 



186 
0 0« 



STO HRKOR 

VARIANCE 

RANGE 



0.005 
0.005 
0.452 



MEDIAN 

KURTOSIS 

MINIMUM 



0.0 PERCENT OF TOTAL 



0.090 
5.232 
0.017 



Table A-9 



SURVEY SAMPLE HISTOGRAM: PROPORTION OF POPULATION BLACK 



06/21/74 



SURVEY SAHPLE OF SCHOOL DISTRICTS 6» 
5ILE NONAMF (CREATION DATS =06/21/74) 

VARiAeia PC8LK PROPORTION OF PQPULATIUN THAT IS BLACK 

CODE 

I. 00 *^t*4t***iM(** ***»»♦»»♦♦♦♦♦♦♦♦♦♦♦ ( 

LEbS THAM ,02 



PAGE 



J*?\ 40.3 POT 



2*00 ( 

.02 10 .0499 



3.00 



4.00 



3.00 



.05 TO .0999 



17) 9.1 PCT 



25) 13.4 PCT 



.L TO .1999 



31) 16.7 PCT 



.2 Ok ABOVE 



3a) 20.4 per 



I« 

0 10 
FREQUENCY 

VALIO OBSERVATIONS - 
HISSING 03SEPVATICNS - 



». I. 

20 



I. 

30 



>. I. 

40 



50' 



..I • 

60 



186 

v) 



. I. 

70 



60 



>.I. 
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Table A-11 

SURVEY SAMPLE HISTOGRAM: PROPORTION OF FAMILIES BELOW POVERTY INCOME 



SURVEY SAMPLE OF SCHOOL DISTRICTS 6. 
PILE NONAME (CRE&TIOf^ DATc = Ob/2l/7^) 

VARIABLE PCPOOR PROPO^TIOiM JF FAMILIES BELOW POVERTY INCoMf 

CODE 

I 

26) 14.0 PCT 



06/21/74 



1*00 ( 

L65>^ THAN .04 



2.00 



3.00 



4.00 



3.00 



.04 TO .0799 



''/*♦«**************♦************** *************** ( 

.OB TO . 1 1999,, 



48) 2f..b PCT 



56) 30. 1 PCT 



««««««««« ««««««« «««««r«««««^«4L««« ( 

.12 TO . 1999 



37) 19»9 PCT 



.2 3R A80V5 



19) 10.2 PCT 



U 

0 10 
FREQUENCY 



20 



30 



.•I. 

40 



50 



>•!« 

oO 



• I. 

70 



BO 



PAGE 14 



90 



<? 



VALID DBS tl<VATlCNS - 
HISSING OBSERVATICNS - 



IBo 

0 



ERLC 



Table A-IZ 

SURVEY SAMPLE: RELATIONSHIPS AMONG SIX VARIABLES IN SAMPLE 



SURVEY SAMPLE OF SCHOOL DISTRICTS 

FILE hONAME (CREATlUN DATE = 06/21/74) 



6« 



06/21/74 



PAGE 









R S 0 N 


C 0 R R E 


L A T I 0 


N C 0 E 




LOGPOP 


LOGENRL 


PCUR8 


PC8LK 


FATCATS 


PCPUOR 


LOGPOP 


UOOOO 
( 0) 
S=0.00l 


0.9654 

{ 136) 

S=iO.00l 


0.6237 
( 186) 
S^O.OOl 


0.2866 
( L86) 
S=0.00l 


o*or>lo 

( 186) 
S=0.489 


-0.1789 
( 186) 
S=0.015 


LOGENRL 


0.9654 
( 186) 
S=0.001 


UOOOO 
( 0) 
S=J.C01 


0.6068 
( 186) 
3=0.001 


0.2559 
( 186) 
S=0.001 


•0.0246 

( 186) 

S=0. 739 


-G. It>j4 
( 136) 
S=0.03/ 


PCUR8 


0,6237 
( 186) 
S=0.001 


0e606d 
( ! S6 ) 
S=0.001 


I. 0000 
( 0) 
S=0.001 


0.0783 
( 186) 
S=0.288 


0.2669 
( 186) 
S=0.001 


-0.2890 
t 136) 

s=o.opi 


PC6LK 


0,2866 
( 186) 
S=0.001 


0.2559 
( iCb) 
S=0.00l 


Oo0783 
( 186) 
S=0.288 


I. 0000 
t 01 
S=0.001 


-0.1883 
( 186) 
S=0.010 


0.4177 
t 186) 
S=0.001 


FATCATS 


0.0510 
( 186) 
S=0. 489 


0.D246 
( 186) 
S = Oa 739 


0. 2669 
( 186) 
^ S = 0.001 


-0. 1883 
( 186) 
S = 0.010 


1.0000 
( 0) 
S=0.001 


-0.4425 

( 186) 

S=0*001 


PC POOR 


-0.17B9 
( 186) 
S = O.Olt) 


-0.1534 
( 186) 
S=0.037 


-0. 2890 
( 186) 
S=0.001 


0.4177 
( 186) 
S^O.OOl 


-0.442:> 
( 186) 
S'0.001 


UOOOO 
( 0) 
S=0.001 



(COEFFICIENT / (CASES) / SIGNIFICANCE) 



(A VALUE OF ^9.ii000 IS PRINTED IF A COEFFICIENT CANNOT BE COMPUTED) 

G 
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.097. From ^efs. 3 and 4 we have total population in 1970, total public school 
enrollment in fall 1969, and number of operating public school districts, from which 
we calculate log of population per district and log of enrollment per district as 9.926 
and 7.777, respectively. We thus have significant overestimates of population, enroll- 
ment, and proportion poor — especially oT population and enrollment. We have an 
underestimate for proportion of families above $25,000 income. Proportions urban 
and black are almost equal to the population estimates. 

Using a multivariate generalization (the T^-statistic) of the t-test, we test for 
equality of vector means between the -sample and population. We find that the 
difference between the two vector means is statistically significant at the .001 level,® 
implying that the survey sample is not very representative of the total population. 
- It appears that the survey sample has significantly larger than average school 
districts. This is at least partly because of selection criteria that eliminated from 
consideration the very small school districts. The smallest in the sample has an 
enrollment of 327." In fact, more than one-third' (36.9 percent) of all public school 
districts in fall 1969 had fewer than 300 students enrolled. But this accounts for only 
1.5 percent of all students enrolled, so log of enrollment per district would increase 
only about .44 if we calculated the population mean only for districts with enroll- 
ments of more than 300 (enrollment figures arc from Table 1 of Ref 3). The differ- 
ence l^etween population and sample means would still be more than seven times 
the standard error of the estimate of the sample mean. In fact, even if we eliminate 
both population and enrollment, and run the test on just the other four vatriables, 
the difference between population and sample vector means is still significant at the 
.001 level." 

We should, of course, recall that our population means are not really means 
over all school districts; they are means calculated from aggregate data. To the 
degree that the means used here reflect the true population means over school 
districts, our survey sample is not very representative of school districts in general. 
Our districts ar,e much larger both in population and school enrollment and have 
proportionately more poor families and fewer families who had incomes over $25,- 
000. The sample seems representative on the whole in terms of proportion black and 
proportion urban, although these variables are both highly skewed in the sample. 

New York City. As pointed out earlier, New York City is not included in the 
above anaiysis.^Nine of the 32 community school districts within New York City are 
included in the survey sample. These districts are compared with city aggregates in 
Table A-13. The nine community districts included in the survey sample appear to 
be representative of city aggregates, at least on the three variables— population of 
school district, public school .enrollment of the district, and proportion of population 
in the district that is black.' 

^ • For p. ~- the vector of population means, x ^ the vector of sample means, S ^ sample covariance 
matrix, N - the number of observations (in this case 186), and p ^ the number of variables (in this case 
six), the ^.statistic is given byT ^ N(x - m)' S '(x - m) Hien (T^ . (N - p)]/(N - Dp = Fp.s dal 
See, for example, Anderson [6]. In this case, 279.66, (T* • (N - p)]/(N - Dp = 45.35, and the .999 
point of F« , 20 = 4.04. 

' The formation is t\ie same as before, except that now p = 4, and jn, x, and S are reduced in size. 
For this case, r ^ 33.18, (r(N ^ pJ]/(N - Dp - 8 16, and the .999 point of F4..20 ■ 4.95. 
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Table A-13 

NEW YORK CITY COMMUNITY SCHOOL DISTRICTS COMPARED WITH CITY AGGREGATES 







Log 


(popu- 






Log (enroll- 


Proportion 


District 


Populat ion 


lation) 


Enrollment 


ment) 


Black 


2 


587,850 


13 


.284 




21,507^ 


9.976 


.036 


3 


285,482 


12 


.562 




21, 856 


9.992 


.281 


4 


127,463 


11 


.756 




21,379. 


9.970 


.382 


7 


161,594 


11 


.993 




30,041 


10.310 


.358 


10 


314 ,782 


12 


.660 




27,693 




. u/y 


11^ 


291,618 


12 


.583 




26,648 


10.190 


.147 


14 


211,853 


12 


.264 


c 


29,739 


10.300- 


.167 


24 


337,814 


12 


.730 




23,085 


10.047 


.043 


32 


114,641 


11 


.650 




21,235 


9.963 


.261 


Sample means 




12 








10.109 


.195 


Minimum 




11 


.650^ 






9.963 


.036 


Maximum 




13 


.284 






10.310 


.382 


NYC totals 


7,892,267 




1, 


116,711 




.211 


NYC district 
















means^ 




12 


.416 






10.460 





^Logs of average district values. 



The Fieldwork Sample 

Census Charactc-ristics. To describe \he fieldwork sample, we 'use the six 
variables used above, with the addition of proportion Spanish language, that is, the 
proportion of the population in the school district for whom Spanish is the major 
' language. Again, New York City will be considered separately. This leaves a sample 
of 22 school districts. 

For the fieldwork sample, we find means for log of population and log of enroll- 
ment of 11.614 and 10.051, corresponding to population and enrollment levels, re- 
spectively, of about 1 10,000 and 23,000. These are both substantially higher than the 
means for the survey sample. Means and medians are again nearly equal, although 
the distributions for both log population and log enrollment are somewhat flatter 
for the fieldwork sample than for the sui^vey sample. The enrollment range is also 
narrower— from 1362, to 650,000. (See Table A44.) 

The fieldwork sites are typically more urban than the survey sample sites. The 
mean proportion of the school district population that is urban is .902. But over 
two-thirds (68.2 percent) of the districts are at least 98 percent urban, and the 
median value is 99.8 percent urban. Only two of the 22 districts are below 60 percent 
urban, compared with 46, or 24.7 percent, of the suivey sample. (See Table A-15.) 

Proportions black and Spanish language are more similar for the two samples. 
Mean proportion black is .130, which is somewhat higher than the .105 mean for the 
survey sample but is within one standard error of the estimate of the (field sample) 
mean. The shapes of the distribtitions for the two samples are almost identical. 
Compare Tables A-8 and A-15. Mean proportions Spanish language are .053 for the 
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Table A-14 

FIELDWORK SAMPLE SUMMARY STATISTICS: LOGS OF SCHOOL DIS'.aIC'^ POPULATION 

AND ENROLLMENT 



FIELD WORK i>AMPLE OF SCHOOL DISTRICTS 
FILE NONAMH (CREATION GATE = 06/2l/7<r» 



06/21/7^ 



PAGE 



VARIABLE 
HEAN 
STO 0£V 
SKEWNESS 
MAXIMUM 



LOGPOP LJG OP SCHOOL DISTRICT POPULATION 

11.614 , STO ERROR 0.3a2 

U791 - VARIANCE 3»209 

0.073 RANGE 6.408 
15.037 



VALID OBSERVATIONS - 
MISSING OBSERVATIONS - 



22 
0 OR 



HEDIAN 

KURTOSIS^ 

MINIMUM 



11.618 
-U082 
8*629 



0.0 PERCtNT OF TOTAL 



VARIABLE 
MEAN 
STD DEV 
SKEWNESS 
MAXIMUM 



LCG6NRL LOG OF SCHOOL DISTRICT ENROLLMENT 

10.051 STO ERROR 0.J61 

1.6S5 VARIANCE 2.874 

0.137 RANiic o»i77 
13.394 



VALID OBSERVATIONS - 
MISSING OBSERVATIONS - 



22 
0 OR 



MEDIAN 

KURTOSIS^ 

MINIMUM 



0.0 PERCENT OF TOTAL 



9- 859 
-1.032 
7#217 



See Table A~4. 
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Table A- 15 

FIELDWORK SAMPLE SUMMARY STATISTICS: PROPORTIONS OF POPULATION URBAN, 
BLACK, AND WITH SPANISH AS MAJOR LANGUAGE 



FIELD MORK SAMPLE OF SCHOOL DISTRICTS 
FILE NONAHE (CREATION C/TE ' 06/21/7^) 



06/21/1^ 



PAGE 



VARUjL^ 
MEAN 
STD DEV 
SKEWNESS 
MAXIMUM 



PCURR 



PROPORTION OF POPULATION THAT IS URBAN 



0*902 
0*178 
-1.752 
1.000 



VALID OBSERVATiCKS - 
MISSING OBSERVATIONS - 



STD ERi^OR 

VARIANCE 

RANGE 



22 
0 OR 



0.038 
0.032 
0.O3S 



iEDlAN 

KURTOSIS* 

MINIMUM 



0.998 
2.057 
0.365 



0.0 PERCENT OF TOTAL 



VARIABLE 
MEAN 
STD DEV 
SKEmNESS 
MAXIMUM 



PCBLK 



PROPORTION OF POPULATION THAT IS BLACK 



0.130 

0. 137 

1. <»33 
0.5'«2 



VALID OBScTiVATIONS - 
MISSING OBSERVATIONS - 



22 
0 OR 



STD ERROR 

VARIANCE 

RANGE 



0.029 
0.019 
0.542 



MEDIAN 
KURTOSIS^ 
M INIMUM 



0.103 
1.956 
0.0 



0.0 PERCENT OF TOTAL 



VARIABLE 
MEAN 
STD DEV 
SKcKNESS 
MAXIMUM 



SPKLAN 



PROPORTION f^ITH SPANISH AS MAJR LANGUAGE 



0.0!>3 
O.C77 
2.1U 
0.310 



VALID OBSERVATIONS - 
MISSING OBSERVATIONS - 



*See Table A-4. 



22 
0 OR 



STO cRRUR 

VARIANCE 

RANGE 



0.016 
0.006 
U.i07 



0.0 PERCENT OF TOTAL 

130 



MEDIAN 

KURTOSIS® 

MINIMUM 



0.020 
"^.044 
0.003 
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fieldwork sample and .060 for the survey sample. The distributions are also very 
similar for proportion Spanish language across the two samples. For the field sam- 
ple, proportion Spanish language ranges from .003 to .310, with a median of .020. 
geven of the 22 have less than 1 percent Spanish language, and three have less than 
percent. (See Table A-15.) 

Family incomes are typically higher in the fieldwork districts, with both a 
higher proportion over $25,000 income and a lower proportion poor, than for the 
survey districts as a whole, but the differences in means are very small. The distribu- 
tions are much flatter and cover a narrower range in the field sample. Proportion 
of families with incomes over $25,000, for example, ranges from .012 to ,127, has a 
mean of .051, a median of .040, and a kurtosis measure almost equal to the value 
for a normal distribution in the field sample. The corresponding values for the 
survey sample are 0 to .359, .046, .036, and a kurtosis measure far from that ol'a 
normal distribution (in the direction of greater peakedness). (See Table A-16.) 

Histograms for the above variables in the fieldwork sample are given in Tables ' 
A-17 through A-23, 

Representativeness, The comparison of vector means betv/cen the survey 
and fieldwork samples is a more straightforward operation than the comparison 
between the survey sample and the population of all school districts. Here we have 
identical data for both samples, so we don't have to estimate population means from 
other sources, and we know the covariances for the population. 

In testing for equality of vector means between the fieldwork and survey sam- 
ples, we find that the differences are not significant at the 5 percent level.® The field 
sites, then, are roughly representative of the whole survey sample, at least on the 
seven variables considered, even though the field sites tend to be substantially larger 
and more urban school districts. 

New YorkCity. Ofthe nine community school districts in New York City that 
are included in the survey sample* one is ia the field sample. It is very close to the 
average of the other eight in population and enrollment, and somewhat lower in 
proportion black. (See Table A-13.) 

* Let ^ =- the vector of means for the survey sample, x ^ the vector of means for the fieldwork sample, 
i. the covariance matrix for the survey sample, N ^ the number of observations (in this case 22), and 
p = the number of variables (in this case 7). Then N(x - m)' ^ *<5c - ^) is distributed as x*p- See, for 
example, Anderson [6). In this case, N(x m)' - *(x - m) = 12.183, the 90 point of x% - 12.0, and the 
.95 point of -141 
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Table A-16 

FIELDWORK SAMPLE SUMMARY STATISTICS: PROPORTIONS OF FAMILIES WITH INCOME 
ABOVE $25,000 AND BELOW POVERTY INCOME 



FIELD WORK SAMPLE OF SCHOOL OISTKICTS 
FILE NONAME ICrEATION DATE = 06/21/7^) 



06/21/74 



PAGE 



VARlABL^E FATCATS PROPORTION OF FAMILIES WITH INCOME ABOVE f ArOOO 



MEAN 
STD OEV 
SKEMN2SS 
MAXIMUM 



0*051 
0.033 
U009 
0*127 



VALIO OBSERVATICNS - 
MISSING OBSERVATIONS - 



22 
0 OR 



STO cRROR 

VARIANCE 

RANGc 



0.007 

o.aol 



0.0 PERCENT OF TOTAL 



HEOIAN 

KURTOSIS^ 

MINIMUM 



0.040 
0.056 
0.0L2 



VARIABLE 
MEAN 
STO OEV 

skewness 

MAXIMUM 



PCPOCR PROPORTION OF FAMlLlcS BELOW POVERTY I NCOA^r 



O.Cd9 
0.053 
l.O'tp 
0.234 



VALIO OBSERVATIONS - 
MISSING OBSERVATIONS - 



22 

0 OR 



STO ERROR 

VARIANCE 

RANGE 



U.Ol L 
0*003 



0.0 PERCENT OF TOTAL 



SEOIAN 0.077 
KURTOSIS^ 0.936 
MINIMUM 0.019 



^See Table A- 4, 
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Table A-17 








FIELDWORK SAMPLE HISTOGRAM: LOG OF SCHOOL 


DISTRICT 


POPULATION 




f^lELO WORK SAMPLE OF SCHOOL DISTRICTS 6* 




06/21/74 


PAGE 8 


FILE NONAHE (CREATION DATE = 00/21/74) 








VARIABLE LOGPOP LOG Of SCHJuL DISTRICT POPULATION 








CODE 

I 

2*00 { 
I 8.5 TO 9.999 








22.7 


PCT 




3*00 ( 4) 18.2 
I 10 TO 10.999 


PCT 






4.00 ***^**» ******************************************** { 

V 11 rO 12.499 


i>J 22.7 


PCT 




:«oo i^*****i^Tk* ***************************************************** ***^*t ************** c 

I 12.3 OR ABOVE 


a> 36.4 PCT 










0 12 3 4 5 
FREQUENCY 


6 


7 


9 I 


VALID OBSEPVAT IC>NS - 22 
MISSING OBSERVATI0>iS - 0 
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Table A-18 

FIELDWORK SAMPLE HISTOGRAM: LOG OF SCHOOL DISTRICT ENROLLMENT 



FIELD «OftK SAHPLE OF SCHOOL DISTRICTS 
FILE NONAME (CREATION GATE = 06/2l/74> 
VARIABLE* LOG^NAL LOG OF SCHCOL DISTRICT E.^ROLLMcNT 
CODE 



6. 



06/21/74 



PAGE 9 



2.00 



3«00 



4.00 



5.00 



I 

****** *********** i 

7 TO B.499 



^^^rnrn************* *************** ****************** i 

8.5 TO 9,499 



22.7 POT 



5i 22.7 PCT 



m*^m ******* **^ *********************** i 

9.5 TO 10.999 



^*^**Mm^9*t^*********** ************* 

U C« ABOVE 



4) 16.2 PCT 



0 I 
FftGOUENCY 

VALID OBS&f«VATICNS - 
KISSING OBSERVATIONS - 



22 
0 



8) 36.4 PCT 
»..I 



i 
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Table 

FIELDWORK SAMPLE HT?:T0GRAI4: PROPORTION OF POPULATION URBAN 



FIELD WORK SAMPLE OF SC^^OOL DISTRICTS b 
FILE NJNAHE (CREATION DATE = 06/21/74) 

VARIABLE PCURB PROPORTIOf^ OF POPULATION THAT IS URBA^^ 

CODE 

1 

2) 9.1 PCT 



06/21/74 



2*00 {, 

.2 TO ,599 



3.00 



4*00 



5.00 



•6 TO .899 



«««««« ( I) 4.5 PCT 

•9 TO .9799 



4) 18.2 PCT 



PAGE 10 



^«i^«j(r «««««««««« mm**** ******* ( 

.98 OR ABOVE 



15) 68.2 PCX 



,0 ^••^•••2* 

" FREQUBNCY 



'VALID OBSERVAT ICNS - 
HISSING. OBSERVATIONS - 



>.I. 

10 



>.I« 

12 



.1. 

14 



>.l. 



..I. 

18 



ERIC 



Table A-20 

FIELDWORK SAMPLE HISTOGRAM: PROPORTION OF POPULATIO BLACK 



FIELD WORK SAMPLE OF SCHOOL DISTRICTS «>• 
FILE NONAME {CREATION CATE = 06/21/74) 

VARIABLE PCBLK PROPORTION OF POPULATION THAT IS BLACK 

CODE 

I .00 ************************************************************* ^ 

LESS Than .02 



06/21/74 



6) 27.3 PCT 



2.00 



.02 TO .0499 



3.00 ***** 



4.00 



5.00 



2) 9.1 PCT 



i 3) 13.6 PCT 



.05 TC .0999 



PAGE II 



♦A*********************************************************** ^ 27.3 PCT 

•I TC .1999 



\ 



♦♦♦♦♦♦♦**************^ 

•2 OR ABOVE 



{ t>j 22.7 p:t 



0 I 
FREQUENCY 

VALID OBSERVATIONS - 
HISSING OBSERVATIONS - 



22 
0 
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Appendix B 



PATTERNS OF ASSOCIATION AMO G PROJECT 
CHARACTERISTIC? 



Section II described separately the three components of project design— re- 
sources, educational method, and implementation strategy. Yet such analytical 
separation somewhat distorts the reality of project design. Some implementation 
strategies were more likely to occur with particular educational methods than with 
others. For example^ we might expect that a high degree of implementation flexibili- 
ty would be associated with projects calling for high levels of classroom organization- 
al change but not for behavioral modification methods. Similarly, it is reasonable 
to expect more cbacentrated funding for projects emphasizing behavioral modifica- 
tion than for projects mostly concerned with enrichment -of many students. This 
appendix describes the patterns of associations or links among the educational 
methods, implementation strategies; project resources and focus, and federal pro- 
grams. 

In particular, the analysis describes the patterns or combinations of project 
resources and focus, implementation strategies, and federal programs that were 
associated in the Rand sample, and could be presumed likely to be more generally 
'associated with the five underlying educational methods. Table B-1 presents meas- 
ures of multivariate association and significance for the relationship between each 
project design choice and the degree of each educational method employed by the 
project. For each design choice, a standardized regression coeflicient (beta weight) 
indicates the extent of the relationship between a project characteristic and an 
educational method when other characteristics are statistically controlled. The 
and F-statistic for each educational method measure the net extent of association 
between the method and the design choices. 

Before describing the substantive results in detail, an overall statistical obser- 
vation is In order. The statistics of Table B-1 suggest, on one hand, that there are 
significant linkages between eciucational method and implementation strategies 
and, on the other hand, that considerable variation remains to project designers in 
choosing the level of educational methods once other strategic choices have been 
made, and vice versa. 



BEHAVIORAL MODIFICATION TECHNIQUES 

The dimension of behavioral modification, liko the other dimensions, ranges 
from very low values (which indicate the presence of no more behavioral modifica- 
tion techniques than are found in a traditional classroom) to very high ones (showing 
that a large component of behavioral modification techniques has been applied to 
the project school). The beta weights "efer to the relative association and covariation 
between the behavioral niodific ' ""n observations and the implementation items 
across the whole range of each scale. A large, significant weight indicates that 
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Table B-1 

PROJECT aiARACTERISTICS ASSOCIATED WITH EDUCATIONAL METHODS 



Project Design Ghoices 



Rcbourres 
Funding 
Students 

Funding per student 



Impleinentation Strategy 
Percentage of planning 
Staff training tine 
Value of mtjet ings 
leacher not participating in 

implementation decision 
Teacher free to alter 

project design 
Materials not <lt vo loped 

local iy 
Project located in special 

imit 

Project covers high and 

clejnenciiry schooKs 
Overall teaching change 

required 
Change in specific teaching 

technique required 
Ext ra ef f ort required 

(principal's view) 



Standardized Re^^ression Coefficients for Educational Methods 



(1) 

Behavlornl 
Modif icat ion 



.10 
-.06, 
.12 



ERLC 



Scliool Characteristics 
Number of students in 
school 

Percentage of black studenti> 

in scliool 
Percentage of Spanish 

students in school 
hiement<.rv school 



.0^ 
.01 
.0^ 
-.01 

.Ob 

-.04 
.04 
13 
-.09 



(2) 



Enrichment 



. 10 
-.01 
-.01 



Classroom 
Organizational 
Change 



-.01 

.06 

.00 

. .00 



.10/ 0/ 



.09 

.i)2 

A 

. 10 

A 

-.19 

A 

'.11 
.0? 

-.0? 

A 

-.14 

.0! 



.07 

-.04 

.03, 
-.14 



. W/. )4 



.10, 
-.16 
-.05 



.17 
.01 

*; 

-.12 

A 

. 10 

-.05 

.o: 
.00 
.o; 

.0 i 

A 

.08 



.OiJ 
-.13 

A 

-.09, 
. 15 



Intensive 
"^radl t ional 
Staffing 



.12 

-OK 

.07 



-00* 
■15, 
.10 

,02 

03 

.11 

A 

. i J 

- i)i 

^.ov 

-.OS 
-.05 



<S) 

Admirustrative 
Change 



-.00 



.14 



.10 
.0? 



. 17/. 14 



•11* 
.13 

02 



.lb 

.03 
.00 

02 

■.07 

A 

-. 12 

A 

'.09 

.{){) 
(18 
-.01 



.13 
.11 



-.02 

.o: 



lO/.o; 



NOre: An asterisk indicates signUi.ance at the .10 level: and a double asterisk ( -V) uuin ate. 

.significance at tl>e .01 level. 

'\U equations are significant for the F-statistic (lb.490). Although the number of l^^;^^)^^": J]; 
509 tL nrof projects is J^;. Ihe classroom level i^^the unit o: analvsis tor thi. table u. allow 
for'classroom and .chool variation in project design and implementation. 
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increasing levels of behavioral modification and the implementation item consist- 
ently occur together. 

The strongest design choice linked to behavioral modification methods was that 
of concentrated, focused" resources for project students. The positive weight for 
funding per student is large (.12) and significant. This reflects the relative concrete- 
ness and visibility of behavioral modification approaches, as well as the fact that 
special centers and materials have a maximum utilization level that cannot be 
stretched. In addition, it appears likely that in schools with a substantial behavioral 
modification innovation, students were clearly either "in" or "out" of the project- 
promoting targeting and focusing of resources on project students. 

The association of total funding levels with behavioral modification was posi- 
tively significant, but differs only slightly from the resource pattern for other meth- 
ods. Overall, large budget projects tended to have more intensive components of 
whatever method they had selected. The significance of funding is thus present at 
the beginning of a project's life, when it can enable major new components of 
schooling to be introduced. 

Particular implementation strategies that were linked to behavioral modifica- 
tion instruments were those appropriate to the simple substitution of one form of 
ajteacher-centered curriculum for another: Local materials development, a special 
unit for all or some project activities, specific teaching technique changes, and a 
minimum of extra required effort were consistent predicates to behavioral modifica- 
tion methods. All had sizable and significant weights, ranging from values of .08 to 
.15 (ignoring the coefficients' signs). Administrative techniques for initiating.change, 
such as planning and meetings, were no more likely to be found in projects using 
behavioral modification instruments than in those not doing so. This probably re- 
flects a tendency to view these projects as "self-winding" technocratic changes that 
take care of their own implementation (see Sec. Ill for a fuller discussion). For 
example, educational technology techniques (which load heavily on the behavioral 
modification factor and can be used as a behavioral modification instrument) may 
be seen as requiring little facilitation to achieve implementation. The consistent 
pattern of implementation for behavioral modification projects was, apparently, to 
concentrate them on selected students and to treat the new educational methods as 
discrete substitutes for specific traditional methods. 

Table B-1 shows that no school characteristics— grades, school size, or ethnicity 
—had a significant relationship with behavioral modification innovations. That is, 
these new treatments were applied across the full spectrum of school types. School 
decisionmakers evidently did not choose behavioral modification methods more 
frequently for any particular group of students — indicating either the treatment's 
adaptability or its possible lack of sensitivity to target group characteristics. 



ENRICHMENT TECHNIQUES 

Although the enrichment dimension was not characterized by increasing con- 
centration of resources for higher levels of enrichment (an extreme enrichment 
program may or may not spend many dollars per student), it shows the same weight 
for its association with aggregate funding as the other dimensions (.10). That is, the 
bigger the total grant, the bigger the amount of enrichment methods is applied. The 
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absence of a concentration effect may be due to the high school focus of these 
projects; the elementary school weight was a negative and very significant -.14. In 
some cases, lump-sum investments may benefit all students in a high school, inde- 
pendent of its size. Perhaps because high school students rotate among classes and 
typically use group facilities such as a speaker, a movie, or a s^oup tour, it appears 
that enrichment projects* resources were not consistently targeted on a select group 
of participants. Other school characteristics (size and ethnicity) were not related to 
the extent of enrichment methods, again suggestmg that broad and inclusive pro- 
jects, to the extent allowed by the resource/treatment mix, were a pattern for 
enrichment. 

Table B-1 indicates that several particular administrative implementation 
strategies were associated with high enrichment innovations. As enrichment compo- 
nents increase, so do planning activities, the value of staff meetings, teacher flexibili- 
ty for adapting the project, and local materials' development. Equally significant 
weights link enrichment with an absence of staff training and an absence of special 
instructional units. Since enrichment activities generally imply an addition to stan- 
dard curricula, usually applied by existing staff using existing methods, this im- 
plementation pattern is readily interpretable. Training was not required since 
teacher behavior was not altered. Nor were new instructional sites or activities 
added. Planning and materials development were required for substantial doses of 
enrichment since project activities had to be consciously designed, and the specific 
activities (e.g., field trips and "community resources" were common in this dimen- 
sion) had to be laid out before students could experience the^. For an innovation 
to be an "enriching" student experience, it must include substantial departures 
from everyday readings, lectures, and discussions— requiring preparation by teach- 
ers. To the extent that teachers* meetings could be successfully initiated and main- 
tained, they were associated with higher levels of enrichment; it appea^-c that teach- 
ers* meetings are fruitful sources of ideas for the design and presentation of enrich- 
ment activities. 

The implementation of enrichment depended oh each teacher's ability to inte- 
grate new materials and substance into existing curricula and workloads. The as- 
sociations between increasing enrichment and teacher flexibility (.10), and an ab- 
sence of specific teaching changes ( - .14), show that enrichment projects were gener- 
ally made to fit individual classroom situations, rather than altering traditional 
approaches. Such change can be one-way, a process in which the project mutates but 
local actors and organizations do not change (see Vol. Ill, Sec. III). In short, the high 
enrichment projects were generally characterized by integration into existing class- 
rooms, but were not a stimulus for new teaching approaches. 



CLASSROOM ORGANIZATIONAL CHANGE METHODS 

Classroom organization innovations tended to be installed in a particular sort 
of setting: in elrmentary schools (beta = .15) that were large (school enrollment 
beta = .08), but in a very small number of individual classrooms (beta for number 
of project students = -.16). Table B-1 shows that smallness of project scope, when 
given as the number of participating students, consistently rises with increasing 
levels of classroom organization treatment. 
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This setting for change— typically one or two classrooms in a sizable elementary 
school— suggests that many innovative classroom projects may rely on experimen- 
talism and voluntarism as the means of introducing new educational methods. 
When very complex and extensive departures from traditional schooling were 
.begun, it seejns that many implementing districts may have confined the change to 
a small (and possibly volunteer) group of project participants, rather than convert- 
ing a whole school to innovative classroom practice at one time. 

Classroom organization changes anchored in voluntarism and selective im- 
plementation appeared to have had a relative absence of black and Spanish-sur- 
named students from participating schools. If schools with substantial minority 
enrollments also tended to center their efforts on traditional "basic" school goals and 
to refrain from free-form educational alternatives, they were perhaps less likely to 
induce parents to volunteer their children — or teachers to volunteer themselves — 
for an unpredictable change from the security of current practice. Complex and 
relatively radical innovations may have proved an unattractive alternative in a 
school immersed in the problems of minority and multilingual education. 

Classroom reorganizatioiis were associated with extensive training; the largest 
weight for any characteristic of classroom projects is that for stafftraining time (.17). 
The training that accompanied a new organization of teacher-child relations in the 
classroom did not appear to bind or constrict teachers* freedom to alter project 
design (the beta for teacher freedom is .10) or their participation in implementation 
decisions (beta = - .12). Training affected implementation by means of linkages that 
were based on a pattern of flexibility, adaptation, teacher control, and very substan- 
tial amounts of extra teacher effort (the last-named effect, with a weight of .08, is 
the only significant, positive association of effort with any dimension of educational 
methods). While the of .16 suggests some but not much overall covariation of 
classroom organization methods with implementation charadteristics, the remarka- 
ble pattern of associations clearly points to the difficult transition facing teachers 
who restructure their classrooms. The insignificance of such administrative tools as 
meetings reflects the relative isolation of elementary teaching, especially when the 
innovating teacher is in one of a small number of *'new** classrooms in a particular 
school.^ 



INTENSIVE TRADITIONAL STAFFING METHODS 

The dimension of intensive traditional staflTmg treatments was strongly as- 
sociated with expenditures, concentration of expenditures, and selection of schools 
with substantially black and Spanish-surnamed students (see col. 4 of Table B-1). 
The traditional approach to educational problems through educational services is 
oflen deemed attractive by minority spokesmen. An additional incentive for minori- 
ties to prefer the intensive traditional approach may have been its use of paraprofes- 
sionals and parent volunteers and councils. 

The dominant implementation strategies for intensive traditional staflTmg were 
those associated with specialization of teaching skills (stafftraining time and special 

' However, preliminary analysis shows that the interaction of stafi'training and meetings was signifi- 
cantly related to high levels of classroom organization changes. See Sec. IV for a discussion of this point 
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units for a project) and the use of prepackaged materials. Specializing and using 
materials created for particular hard-to-solve learning problems are both well suited 
to individualized, prescriptive, or remedial teaching tasks. This top-down struc- 
tured, training-intensive approach may have been effective in bringing the intensive 
traditional treatment to bear on target students, especially when personnel (such as 
paraprofessionals or parent volunteers) were not allowed or qualified to undertake 
major whole-child interventions. 

The value of meetings among project staff, as shown in Table B-1, may take its 
importance from the need to integrate the activities of diverse, specialized instruc- 
tional personnel involved in intensive traditional staffing. 



ADMINISTRATIVE CHANGE METHODS 

The dimension of administrative change methods is somewhat difficult to cha- 
racterize; it includes heavy loading for such techniques as behavioral objectives, new 
management techniques, and new curricula. It appears to emphasize vigorous, broad 
policies for attacking persistent problems, with a strong component of administra- 
tive involvement and supervision. 

Table B-1 indicates that high administratively directed change projects, such as 
the classroom organization approach, were characterized by a focus on a relatively 
small number of students in a relatively large school. However, implementation of 
the change strategy tak^artompletely different tack: Instead of training staff 
members, administrative! change districts may have expended their project re- 
sources on planning by administrators; instead of freeing teachers to adapt projects 
to their own classrooms, ^;teachers were enjoined not to make alterations in the 
project's design. Material^ tended to be developed locally, but not necessarily by 
teachers. | 

In summary, projects ihigh on the administrative change dimension were as- 
sociated with top-down, administratively enforced changes— which, however, were 
carried out by individual teWhers (the weight for special project units is - .09). Such 
an approach may be suited to the quick imposition of a new remedial reading and 
testing program, for instai^e; in cases where extensive administrative preparation 
but no classroom modificatiions were appropriate, administrative change methods 
and this pattern of implementation strategies could fit together. 

1 ■ ■ 



Appendix C 



SCHOOL DISTRICTS* PROPENSITY TO ADOPT 
INNOVATIONS 



Our field experience strongly suggested that innovativeness depended on char- 
acteristics of the school district. Some school districts seemed more likely to innovate 
and to produce successful innovations than other districts. The literature on educa- 
tional innovation often ignores the institutional setting, and studies that do analyze 
organizational aspects usually focus on the school but neglect the district. 

This appendix presents a preliminary analysis of the propensity of school dis- 
tricts to adopt educational innovations. Aside from the importance of this topic in 
its own right, the analysis accomplishes two objectives. First, we suspect that the 
factors that affect the district's propensity to innovate will also affect initiation and 
continuation of special projects. Thus, this analysis is a source of hypotheses for Sec. 
IV. Second, the **innovativeness scale" developed here can be used directly in the 
analysis of continuation. Indeed, the scale is correlated .32 with continuation. How- 
ever, due to the intercorrelations among the explanatory factors, innovativeness 
was not analyzed fully in Sec IV. 



MEASURING INNOVATIVENESS 

Innovativeness is an elusive concept. Not only is there no agreed-on definition 
of innovativeness in the literature, but operational measurements differ widely. We 
view innovation as a change process involving various stages. Different concepts of 
innovation emphasize different stages in the change process. One concept deals with 
the invention of new techniques, strategies, or arrangements. The invention of 
educational strategies, methods, or technology is not the focus of this study. Rather 
we are concerned with the change processes initiated by the adoption of projects or 
programs that are new relative to the adopting school district (or units within the 
school district). Alternative definitions of innovation focus on two other aspects of 
change processes. First, innovation has been defined as the successful implementa- 
tion of a project or program that is new in the district. Second, innovation has also 
been taken to mean that the introduction of a project or program produces a presum- 
ably improved outcome. Without begging the question of the extent to which adop- 
tion of a new program implies either its full implementation or significant outcomes, 
we shall measure the propensity of school districts to adopt ^^innovations" by sum- 
ming up the number of widely discussed edu ational innovations tried by the district 
in the last decade. 

Table Ol presents a list of 21 ^^educational innovations."* Each superintendent 
of the school districts in our sample (n - 194) was asked to indicate for each 

* Superintendenta were also asked about the adoption of bilingiial programs However, since such 
programs are only adopted m LEAs having significant non-English-speaking pupils, they are dissimilar 
from other educational innovations on the list used and hence were deleted. 
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Table C-1 

SUPERINTENDENTS* RESPONSES TO LIST OF INNOVATIONS IN THE SCHOOL DISTRICT 



Question: Here is a list of eduoatioml innovations tint have been tried out 
in some school districts in the last decade. For each innovation, please circle 
the appropriate code to indicate whether it MS never tried, tried but not nou) 
incorporated^ or has been incorporated into current practice in your district. 



Innovation 



Never 
Tried 
(X) 



Tried in 
District But 

Not Now 
Incorporated 

a) 



Incorporated 
into Current 
District 
Practice 
(%) 



Number 
Not 
Answering 



Programmed learning 
Extended school year 
Extended field trips 
Team teaching 

Nongraded or ungraded classrooms 

Flexible scheduling 

PSSC^ Phyfjics 

Typing in elementary 

Community school 

Work/study program 

Teacher corps 

Student exchange 

Educational TV 

Simulation or gaming 

Individualized instruction (method 

and/or materials) 
Open classrooms 
Program budgeting (PPBS) 
Behavioral objectives 
Alternative school 
Special classes for the gifted 
Needs assessment 



19.21 
72.88 
17.77 

5.11 
15.08 
20.00 
29.82 
57.71 
54.71 

9.09 
71.67 
30.81 
13.55 
26.90 

0.56 
23.03 
52.80 

7.91 
46.86 
22.15 
10.11 



28.25 
3.38 
10.55 
5.68 
10.05 
15.42 
14.03 
18.85 



65 
41 



8.67 
18.60 
15.25 
19.29 

5.08 
8.43 
13.48 
14.12 
7.43 
17.04 
15.17 



52.54 
23.72 
71.66 
89.20 
74.86 
64.57 
56.14 
23.42 
37.65 
87.50 
19.65 
50.58 
71.18 
53.80 

94.35 
68.54 
33.70 
77.97 
45.71 
60.79 
74.72 



17 
17 
14 
18 
15 
19 
23 
19 
24 
18 
21 
22 
17 
23 

17 
16 
16 
17 
19 
18 
16 



NOTE: Number of observations = 194. 
^Physical Sciences Study Committee. 



innovation whether the school district had tried the new program and .whether the 
program was currently incorporated into district practice. If fully implemented, 
some of these educational innovations (e.g., the extended school year) would require 
extensive changes in the administrative life of a school district; others, such as 
educational television, imply fewer organizational changes. Table Ol indicates that 
such practices as team teaching, work/study program, and individualized instruc- 
tion have been generally adopted by the districts in this sample, whereas other 
practices such as extended school year, teacher corps, and PPBS have not been as 
widely adopted.^ The question this section asks is what aspects of the characteristics 

» Responses to whether the pro-ams were ''tried in the district but not now incorporated" are 
ambiguous because a ''yes" response could indicate either that the project was tried and rejected or is 
being tried and has not yet been incorporated. To avoid errors due to this ambiguity, the analysis 
categorizes the responses for each innovation into either "never tried" or "tried." 

1'18 , 
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of school districts land how they vary) exphiin the differences in their propensity to 
adopt innovations of the type indicated in Table C-1. 

The analysis could proceed by considering each innovation separately as a func- 
tion of theoretically plausible characteristics of the district and then comparing the 
results for each innovation. Because this procedure would be costly and might tend 
to be dominated by the specific substance of Ihe list of educational innovations used 
here, we approach the measurement of the dependent variable by aggregating the 
individual measures. In particular, we employed the following five scales: 

1. An unweighted sum of the responses to all the educational innovations 
where a zero score was given if the district had not tried the innovation and 
a score of one otherwise. 

2. An unweighted sum of the responses to those innovations that primarily 
involve the student in the classroom and do not imply major administrative 
changes in the school or school district organization.^ 

3. An unweighted sum of the responses to those innovations that imply ad- 
ministrative or organizational changes in the school or school district.^ 

4. A weighted sum of the responses to those innovations that imply adminis- 
trative or organizational changes in the school or school district.^ 

5. A weighted sum of the responses to all innovations using the weights of 
scale four. 

The point of using these alternative scaling procedures is to explore several 
problems of validity in the construction of an innovation index. First summing the 
various items tends to mask overly strong effects of the specific substance of each 
item. However, such aggregation necessarily makes the index abstract and thus 
should be interpreted as the propensity to adopt current educational innovations. 
Second^ since the aggregation of all items might lose **too much'* of the substance 
of the innovations, scales two and three separate the student-class-oriented innova- 
tions that do not imply administrative changes from those that do involve adminis- 
trative alterations. Third, the various innovations undoubtedly differ in the ease 
with which they might be adopted; an equal weighting scheme assumes away these 
differences. Scales four and five represent a preliminary effort to weight the innova- 
tions and thus enable us to examine the sensitivity of the results to an equal 
weighting assumption. Table C-2 presents the statistical characteristics of the five 
innovativeness scales. 



EXPLAINING SCHOOL DISTRICT INNOVATIVENESS 

In "explaining" innovativeness here, we will not deal directly with internal 
processes or decisions within the school districts. We will try instead to explain the 
difference^ in the propensity to adopt innovations in terms of theoretically plausible 

' The items included are programmed learnmg, extended field trips» FSSC physics, student exchange, 
e<lucationa! TV, simulation or gammg, individualized instruction, and special classes for the gifled 

* The items included are extended school year, team teaching, nongraded or ungraded classrooms, 
flexible scheduling, community school, work/study program, teacher corps, open classrooms, PPBS, 
behavioral objectives, alternative school, and needs assessment 

* The following weights were used, three for alternative school, two for each of open classroom, 
nongraded or ungraded classrooms, and team tet^ing^ and one for the remainmg administra' e items. 
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Table C-2 

STATISTICAL CHARACTERISTICS OF INWATIVENESS SCALES 



Scale 


Mean 


Median 


Standard 
Deviation 


Range 


Sum of all innovations 












(unweighted) 


13.9 


13.9 


3.5 


0- 


20 


Sum of student-class 












innovations 


-v6.8 


7.1 


1.8 


0- 


•9 


Sum of administrative 












innovations 


7.1 


7.2 


2.2 


0- 


-11 


Sum of administrative 












innovations 












(weighted) x 


^^9.8 


i9.9 ^ 


; J. -2 


0- 


-15 


Sum of all innovations 












(weighted) tf' 




^& 


4.5 


0- 


-24 



characteristics of the district. Considerable empirical literature about the diffusion 
of innovation, particularly in the fields of agriculture, medicine, public bureaucra- 
cies, and economic firms, suggests that size, wealth, and the availability of resources 
may be related to the propensity of organizations to adopt innovations.* That is, 
larger and wealthier organizations often appear to adopt more innovations. This 
finding may seem contrary to our intuitive feeling that large organizations are 
"conservative" in nature; for school districts, the finding is perhaps even more 
counter-intuitive. Nonetheless, ^ the following analysis shows, factors related to 
size and wealth strongly affect school district innovativeness. 

Table C-3 presents the- results of a statistical analysis of the school district's 
propensity to adopt considered innovations, as determined by a variety of character- 
istics of the district. In particular, using ordinary least sauares estimation proce- 
dures, the independent variables used to explain the variation in school district 
innovativeness represent five groups of factors that, on a priori grounds, might affect 
innovativeness. 

. The first group consists oftwo measures of size: total LEA enrollment and 
number of students per school. • 

• The second group consists of four measures of the district's financial situa- 
tion: the expenditure per pupil of the district measured in ferms of the 
district's deviation from the state's average; the average expenditure per 
pupil of school districts in the district's state; the general financial situa- 
tion of the district as assessed by the superintendent; and whether the 
district (according to the superintendent) has been forced to cut back on 
programs because of financial circumstances. 

• The third group r«^lates to the source of the district's revenues: the per- 
centage of the dif trict's revenue derived from state or federal sources 
measured in terms of the district's deviation from the state average. 

« The findings in the literature about the importance and nature of the effects of size are somewhat 
mixed and in dispute. 
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(st.i!u',ir 1 error) (probdhi 1 U\ v.ihu) 


Independent V«irlables 


All 
Innovat Ions 


Ciabs 
Innovat ions 


Adninistrative 
innovations 


Administrat i ve 
Wei>;ht 


Ml Wei Kb I 


Ol^jcrict enrollment (log) 
Pupils per school (resi'iudl) 


A« 

i.4y 

(.168) (.00) 

-1.85 
(.515) (.00) 


A A 

.09 7 
( 101) (.00) 

-1.03 
(.M2) (.00) 


. 790*' 
(.113) (.uO) 

X A 

-.827 
(.348) (.02) 


A>f 

1. lu 

(.KS2) ( .Oi)) 

A '< 

-1. 19 
(.561> (.0'.) 


1. SU^ 
(.2.'6) (.00) 

-^.22 
(.697) (.nn) 


■ r-- 

L* penditures per pupil/state 
average 

State average expenditure 
per papil 

Adequac> of district linancial 
*>icuatiot\ 

cutbacks in programs 


.183 
(.667) (.78) 

- 0005 
(.001) (.73) 

.396^' 
(..240) (.10) 

AA 

-.890 
(.416) (.03) 


-.073 
(.402) (.86) 

-.000 i 
(.001) ( 71) 

.244 
(.1^-4) (.09) 

-.250 
(.251) (. i2) 


.256 
(.448) (.5?) 

-.000 7 
( 001) (.40) 

.152 
(.161) (.35) 

-.640 

(.279) (.02) 


.t>6 . 
(.722) (.36) 

- . 00 Ou 
(.001) (. 8) 

. 345 
(.260) (.187) 

7.827'' 
(.450) (.07) 


-.0001 
(.002) 1.95) 

. 58h" 
(.3.3) (.07) 

-1.08* 
(.56) (.06) 


^entage of revenue trora sitit*^/ 
state average 

btate average percentage of revenue 
trom state 

Percentage of revenue froc federa^/ 
siaCe average 

State average percentage ut revenue 
froo federal 


*A 

- 1 .' 

(43.3) (.01) 

.036^ 
(.021) (.09) 

6.60 
(16. Ji) (.70) 

-. l('4 
(.969) (.?^)' 


A 

(29.1) (.06) 

.005 
(.013) (.67) 

3.96 
(10.1) 1 .'0; 

- ,015 
'.(.058) (.80) 


A A 

(32.4) (.05) 

AA 

.0 30 
(.014) (.0^) 

2.64 
(11.3) (.81) 

-.089 
(.065) (.18) 


-65 8 

(52.2) (.21) 

.030 
(.023) (.19) 

16.1 
(18.2) (.38) 

-.057 
(.105) (.59) 


A 

-1^2. 

(65. (.06) 

.0^6 
(.028) (.21) 

20.0 
(22.6) (.38) 

-.072 
(.130) (.58) 


Percentage of facllies with income 
$25,000 

Percentage of poor families predicted 
frees minority 

Rural 


12.7 
(5.55) (.02) 

AA 

-11.3 

(5. W) -(.03) 

/ -.858 
(.687) (.21) 


6.93 
(3.35) (.04) 

AA 

-8.35 
(3.08; (.01) 

. -.159 
(.414) (.70) 


5.78 
(3.37) (.12) 

-2.91 

(3.43) (.39) 

-.698 
(.461) (.13) 


11. 3 
(6.01) (.06) 

-5.30 
(5.54/ (.34) 

A 

-1.31 
(.743) (.08) 


AM 

18.2 
(7.47) (.02) 

AA 

-n.6 

( D . 00; V . 03 ; 

-1.47 
(.^>24) (.11) 


Tenure of superintendent 

Years of superintendent's previous 
experience Ir. district 


AA 

.131 

(.038) (.ou; 

.031 
(.024) (.21) 


AA 

.070 
(.023) (.00) 

.020 
(.015) (.18) 


AA 

.062 
(.026) (.02) 

.011 
(.016) (.51) 


A 

.073 
(.041) (.08) 

.0J9 
(.026) (.48) 


AA 

.143 
(.51) (.01) 

.038 
(.033) (.24) 


2 

R*^ (corrected) 
Correlation coefficient 
Degrees of freedom 
Range of dependent variable 


.58 
.79 

134 
0-20 


.44 
.70 
134 
0-9 


.49 
.74 
134 
0-11 


.41 
.69 

134 
0-1^ 


54 
.76 
134 
0-24 



CO 



NOTb: An astifj^k * ) indicates si^niMtaiue at the 10 U'vei;- )nd <i doubU' asterisk 



5>igni t'lv anco at th' ,0:> level 
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• The fourth group consists of three measures of the socioeconomic-ethnic 
characteristics of the community in which the school district is embedded: 
the percentage of families in the community with incomes over $25,000; the 
percentage of families in the community with incomes under the poverty 
level combined with the percentage of families from minority groups (black 
or Spanish-speaking); and whether the district is in a rural area. 

. The' final variable is the tenure of the district's current superintendent. 

In the course of analyzing the results, we discuss the meaning of these variables 
more fully and interpret their theoretical significance. 

Table 03 displays the regression coefficients from each of the independent 
variables along with their standard errors and probability values. The same struc- 
tural equation is used for the five measures of propensity to adopt innovations 
previously discussed. R^ the proportion of the variation (adjusted for the degrees of 
freedom used in the estimation) explained by the independent variables, is indicated 
below each column. 

The variable with the largest effect on the propensity to adopt innovations is the 
size of a school district as measured by enrollment.^ Controlling for other factors, 
it accounts for approximately 30 percent of the variance in the dependent variables. 

Why do larger schooldistricts tend to adopt more innovations? One reason is 
tiiat large school districts with numerous and diverse schoo.. have more opportuni- 
ties to innovate.® 

A related theoretically plausible explanation is that larger school districts have 
more "organizational slack" than smaller districts. Organizational slack in the 
context of the school district can occur in several ways. Larger school districts have 
larger operating budgets and a greater flexibility to direct funds within that budget. 
Perhaps even more important, size may allow political flexibility. That is, the moti- 
vations for school districts to adopt innovations are onmnlex and mixed. For any of 
a variety of specific reasons, the district may feel either a positive desire or a reactive 
net^d to introduce new programs. Moreover, although district decisionmakers may 
be risk-adverse in the sense that they may be more concerned with avoiding failures 
than promoting change, the consequences of failure of a project would have fewer 
and more diffuse political repercussions in a hrge district than a project with the 
same scope in a smali district. In political terms, bigger districts can better afford 
to experiment than smaller districts. 

Several impl itions for the prospects of particular innovations follow from this 
size effect. For a project of the same relative "scope," we would expect less pressure 
from above in a larger district than in a smaller one. " ^3 do not have plausible 
theoretical reasons to suggest what the direction of the effect of such pressure on 
the success of a project might be; we take this question to an empirical issue. 
However, it is reasonable to h / >othesize that, assuming equal success of an experi- 
mental project of the same relative scope, the larger district would he less likely to 
incorporate a project, that is, propagate it throughout the district on a regularized 

' Since the distnbution of enrollment is highly skewed a logarithmir transformation of the enroll- 
ment was employed leading to a much closer to normal distribution The regression coefficient for log 
enrollment should be interpreted as there being an average increa:*e of 1.56 innovations for every change 
of one in the natural Ic^arithm. 

• Future anftlysis will consider measures of the propensity to innovate that are not sensitive to the 
accumulated number of innovations. 
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basis. For the smaller district, the lack of political slack means that adoption) is 
equivalent to placing one's bets. For the larger district, the availability of political 
slack means the adoption amounts to experimenting; when the time comes to incor- 
porate, the pressure is to spread the innovation throughout the district. Such propa- 
gation creates severe political risks. 

We have deliberately glossed over the phrase '^relative scope" ofa project. A 
project involving the same absolute Jevel of resources (teachers, materials, and 
expenditures) clearly has different economic consequences in a small district than 
in a larger district. In particular, the same'^project in the small district can be 
expected to have a larger opportunity cost than in a large district. Such opportunity 
costs cannot be calculated, but nonetheless they play an important role in district 
decisions. Moreover, identical projects may have more political visibility in a small 
district than in a large district. 

Another dimension of district size also affects innovation. Districts that have 
the same overall enrollment may differ in the number and size of the schools within 
the district. To capture part of this important organizational difference among 
school districts, we used as an explanatory variable the enrollment density for the 
district, that is, the average number of students per school in the district.® Density 
decreases innovation, other things being equal. Although density's effect on innova- 
tion is about one-third as great^as that of enrollment, it is highly significant (see 
Table C-3). 

Several plausible explanations of the importance of density can be offered. Per- 
haps the most compelling theoretical reasons are based on organizational slack. The 
more dense the district (and thus the fewer the number of schools for the same 
enrollment), the less slack exists both in economic and political terms. 

In addition to size-related characteristics, the effects of "wealth" (controlling for 
size and other wealth-relatecf characteristics) can be expected to affect the propensi- 
ty to adopt innovations. Measuring the wealth of a school district is an extraor- 
dinarily complex task. Not only is it difficult to conceptualize what the appropriate 
measures of wealth should be, but gathering the appropriate information from 
school officials, who cannot be expected to keep their financial records in a theoreti- 
cally useful way, is, at best, uncertain. To cope with this situation, We used surrogate 
measures involving expenditure per pupil and the district's general financial situa- 
tion. 

Expenditure per pupil varies considerably across individual school districts 
throughout the country. However, part of this variation is due to differences among 
the states in such areas as state policies and regional wage rates. Since our sample 
of school districts was picked by a first-stage State selection, we need to control the 
effect of the state on the expenditures of school districts within the ctaU?, Therefore, 
the regression includes both deviation of each school from the state average expendi- 
ture per pupil and th absolute value of the state average. Neither of these variables 
was significantly related to innovativeness. 

That the state average expenditure per pupil fails to be significant is not sur- 
prising. Much of this difference in averages among states is due to such costs as 

• The enrollment density increases with larger enrollment. (Overall, (log) enrollment is correlated 
485 with enrollment per school.) Since we are interested in the elFect of density independent of enroll- 
i-nent. the variable used m the regression is the residual of (log) pupils per school regressed on (log) 
enrollment. ^ 
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teacher salaries and operating and maintenance costs of districts rather than to the 
type of additional expenditures that might result in economic slack.'^ 

The lack of significance of the relative expenditure per pupil is more surprising 
—at least at first thought. The reason why it is not statistically significant may be 
that different levels of per pupil expenditure reflect two conflicting tendencies. 
Insofar -as school dilstricts spend more money per pupil than other districts (ir the 
same state) in order to deal with problems such as -compensating for children from 
poor or minority families, then these higher relative expenditures dc not represent 
slack. But higher relative expenditure per pupil can also reflect a greater local 
wealth base of the district and political pressures from the wealthier members of the 
community; in this case, one does expect slack and an impetus toward innovation. 
In short, the differences in relative expenditure per pupil arise fronr conflicting 
sources and thus cause expenditure per pupil not to be significantly related to 
innovativeness unless these other sources are taken into account." 

Whereas expenditure per pupil has little effect, two other direct bu. "noisy" 
measures of the financial situation do affect the propensity to adopt innovations. We 
asked each superintendent to indicate what the present financial situation in the 
district was and whether the district had been forced to cut back on programs.'^ 
Because both of these variables rely on the subjective judgment of superintendents, 
their validity needs to be questioned. Nonetheless, it seems reasonable to interpret 
the superintendent's answer to the financial situation for carrying out needed educa- 
tional programs as a surrogate for the extent of economic slack (excess over needed 
funds) available in the district. In aiiy event, the better the financial situation, the 
higher the propensity of districts to innovate. Half of the superintendents in our 
sample said they had to cut back on programs due to financial shortages. Their 

For example, the rangeoftheaverageofthesalariesoftheinstructionalstaffforth^ 
stale (New York) to the lowest paying state (Arkansas) - -ur sample is ^^^'P^^to $6715 (K««^ 
Division. National Education Association, Rankings of the States, 1972, Research Report 1972.R1, 1972.) 

' » The data support ihv above explanation for the lack of significance of the relative expenditure per 
pupil in the following way: The zere^rder correlauon between relative expenditure and the innoyative. 
ness scale is approximately .2, but expenditure per p«'pil also has a zeroorder corre ation of 13 with the 
percentage of families in the district who are blacK or SpapJ^h-speaiving and .19 and .12 with the 
percentage of families with incomes over $25,000 and the overall financial situation Of the district as 
reported by the 3uperint€ndent. respectively. The analysis includes and controls for all of these variables 
(as well as the others indicated in Table C3). in which event expenditure has a positive but not significant 
effect on innovativeness with a partial correlation under .1. 

The questions and the marginal results were a follows: 

How do you view the present financial situation in your district! V/ould you say your budget is 
more than adequate, adequate, barely adequate, or inadequate to carry out needed educational 
programs! 

More than adequate .... - . ■ 29.8% 

Adequate 26.7% 

Barely adequate 39.8% 

Inadequate 3.7% 

Number of responses = 194; no answers = 5. 

Has your district been forced to cut back on programs in the last few yean as a result of financial 
shortages! 

Yes 49.0% 

ATo 51.0% 



Number of responses = 194. no answers - 2. 
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districts, which were on the financial margin ai. ' were less likely to have economic 
slack, were less likely to adopt innovations. 

The above findings can be further clarified by examining the analysis of class- 
type innovations versus administrative innovations. Table C-3 shows that for the 
subset of clas^-type innpvations, the general financial situation is significant (indeed 
more so than for the innovativeness scale including all innovations), whereas cut- 
back fails to be significant. The opposite result holds for administrative-type innova- 
tions. A plausible interpretation of these results is that districts having economic 
slack are more likely to adopt class-student-type innovation^ and districts operating 
at the financial margins are less likely to adopt administrative-type innovations. 

One important" implication of the above findings for particular innovative pro- 
jects is clear: We wo"uld expect a district having financial slack to be more likely 
to continue a class-student-type project on its own funds after initial federal funding 
is completed than it would an administrative-type project. 

The next group of variables is concerned with the source of financial support for 
school districts. There is considerable variability among school districts in the extent 
to which their revenues come from local government, the state, or from the federal 
government. A major aspect of this variability is related to the differences among 
states in terms of their wealth, their demographic characteristics, and their state 
policies toward financing education. For example, the percentage of revenue for 
schools from state government varies in our sample from a higli of 68.7 (North 
Carolina) to a low of 21.7 (Massachusetts). To correct for state variation, the analysis 
uses both the absolute value of the state average percentage and the relative devia- 
tion of the school district from its state average. 

The results of the analysis shown in Table 03 indicate that differences among 
school districts in the percentage of federal funds they receive vis-a-vis revenue from 
state and local governments does not significantly affect the propensity to adopt 
innovations. This finding is understandable when one considers the "entitlement"' 
or grants-in-aid basis for federal funding. 

Differences among school districts in the percentage of revenue derived from 
the state government does significantly affect innovativeness. The higher the per- 
centage of revenue from the state relative to other school districts within the state, 
the lower the propensity to innovate. Two hypotheses may explain this result. First, 
incrementally more state money may be going to districts that need the funds in 
order to deal with their financial problems. Thus, the relative deviation of a district 
from the state average in the percentage of funds received from the state is negative- 
ly correlated with both the relative expenditure per pupil (-.31) and the general 
financial situation of the district (-.24); it is also negatively related (-.31) to a 
measure that in part reflects the wealth base of the community, the percentage of 
families with income over $25,000. Moreover, the relative percentage of revenue 
received from the state shows a high negative correlation (-.89) with the relative 
percentage of revenue received from the local government. Second, state monies 
may be more tied down than either local or federal funds. Both of these hypotheses 
work in the same direction toward reducing the amount of slapic for the school 
distriCv. 

The next group of variables, representing demographic characteristics of the 
community, are importantly related to innovativeness. The percentage of families 
with income of at least $25,000 increases innovativeness, whereas the percentage of 
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families with income below the poverty level, combined with minority, decreases 
innovativeness. Rural decreases innovativeness as anticipated, although the result 
fails to be significant. 

The final group of variables refers to a factor that analysts of school systems 
believe strongly influences the policies of the school district— the superintendent. 
The tenure of a superintendent (whether measured in terms of numbers of years as 
superintendent or as a dummy, variable for three or fewer years or more than three 
years) increases innovativeness. This result holds up even when controlling for the 
mobility and the past experience of the superintendent. A hypothesis explaining this 
result is a political-organizational one. innovations in school districts generally 
come incrementally and involve the ability of the superintendent to use the political 
slack in the system. More experienced superintendents know how to gjanipulate 
their system better.'^ 

This result may be affected by a measurement error in the dependent variable. It is possible that 
newer superintendents would not have knowledge of all the innovations in the district during the past 
ten years and therefore would underreport "older*' innovations This bias is minimal in the data because 
most school superintendents had worked their way up in the same school system. 



ERiC 



